{
    "file_diffs": [
        {
            "old_file_content": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"\n\"\"\"\n\nfrom collections import namedtuple\n\nimport json\nimport threading\nfrom fasteners import InterProcessLock\nfrom functools import lru_cache\nimport datalad\nfrom datalad.consts import (\n    DATASET_CONFIG_FILE,\n)\nfrom datalad.cmd import (\n    GitWitlessRunner,\n    StdOutErrCapture,\n)\nfrom datalad.dochelpers import exc_str\n\nimport re\nimport os\nfrom pathlib import Path\n\nimport logging\nlgr = logging.getLogger('datalad.config')\n\ncfg_kv_regex = re.compile(r'(^.*)\\n(.*)$', flags=re.MULTILINE)\ncfg_section_regex = re.compile(r'(.*)\\.[^.]+')\ncfg_sectionoption_regex = re.compile(r'(.*)\\.([^.]+)')\n\n\n_where_reload_doc = \"\"\"\n        where : {'dataset', 'local', 'global', 'override'}, optional\n          Indicator which configuration file to modify. 'dataset' indicates the\n          persistent configuration in .datalad/config of a dataset; 'local'\n          the configuration of a dataset's Git repository in .git/config;\n          'global' refers to the general configuration that is not specific to\n          a single repository (usually in $USER/.gitconfig); 'override'\n          limits the modification to the ConfigManager instance, and the\n          assigned value overrides any setting from any other source.\n        reload : bool\n          Flag whether to reload the configuration from file(s) after\n          modification. This can be disable to make multiple sequential\n          modifications slightly more efficient.\"\"\".lstrip()\n\n# Selection of os.stat_result fields we care to collect/compare to judge\n# on either file has changed to warrant reload of configuration.\n_stat_result = namedtuple('_stat_result', 'st_ino st_size st_ctime st_mtime')\n\n\n# we cannot import external_versions here, as the cfg comes before anything\n# and we would have circular imports\n@lru_cache()\ndef get_git_version(runner=None):\n    \"\"\"Return version of available git\"\"\"\n    runner = runner or GitWitlessRunner()\n    return runner.run('git version'.split(),\n                      protocol=StdOutErrCapture)['stdout'].split()[2]\n\n\ndef _where_reload(obj):\n    \"\"\"Helper decorator to simplify providing repetitive docstring\"\"\"\n    obj.__doc__ = obj.__doc__ % _where_reload_doc\n    return obj\n\n\n# TODO document and make \"public\" (used in GitRepo too)\ndef _parse_gitconfig_dump(dump, cwd=None, multi_value=True):\n    \"\"\"Parse a dump-string from `git config -z --list`\n\n    Parameters\n    ----------\n    dump : str\n      Null-byte separated output\n    cwd : path-like, optional\n      Use this path to convert relative paths for origin reports\n      into absolute paths\n    multi_value : bool, optional\n      If True, report values from multiple specifications of the\n      same key as a tuple of values assigned to this key. Otherwise,\n      the last configuration is reported.\n    \"\"\"\n    dct = {}\n    fileset = set()\n    for line in dump.split('\\0'):\n        if not line:\n            continue\n        if line.startswith('file:'):\n            # origin line\n            fname = Path(line[5:])\n            if not fname.is_absolute():\n                fname = Path(cwd) / fname if cwd else Path.cwd() / fname\n            fileset.add(fname)\n            continue\n        if line.startswith('command line:'):\n            # nothing we could handle\n            continue\n        kv_match = cfg_kv_regex.match(line)\n        if kv_match:\n            k, v = kv_match.groups()\n        else:\n            # could be just a key without = value, which git treats as True\n            # if asked for a bool\n            k, v = line, None\n        present_v = dct.get(k, None)\n        if present_v is None or not multi_value:\n            dct[k] = v\n        else:\n            if isinstance(present_v, tuple):\n                dct[k] = present_v + (v,)\n            else:\n                dct[k] = (present_v, v)\n    return dct, fileset\n\n\ndef _update_from_env(store):\n    overrides = {}\n    dct = {}\n    for k in os.environ:\n        if k == \"DATALAD_CONFIG_OVERRIDES\":\n            try:\n                overrides = json.loads(os.environ[k])\n            except json.decoder.JSONDecodeError as exc:\n                lgr.warning(\"Failed to load DATALAD_CONFIG_OVERRIDES: %s\",\n                            exc)\n        elif k.startswith('DATALAD_'):\n            dct[k.replace('__', '-').replace('_', '.').lower()] = os.environ[k]\n    if overrides:\n        store.update(overrides)\n    store.update(dct)\n\n\ndef anything2bool(val):\n    if hasattr(val, 'lower'):\n        val = val.lower()\n    if val in {\"off\", \"no\", \"false\", \"0\"} or not bool(val):\n        return False\n    elif val in {\"on\", \"yes\", \"true\", True} \\\n            or (hasattr(val, 'isdigit') and val.isdigit() and int(val)) \\\n            or isinstance(val, int) and val:\n        return True\n    else:\n        raise TypeError(\n            \"Got value %s which could not be interpreted as a boolean\"\n            % repr(val))\n\n\nclass ConfigManager(object):\n    \"\"\"Thin wrapper around `git-config` with support for a dataset configuration.\n\n    The general idea is to have an object that is primarily used to read/query\n    configuration option.  Upon creation, current configuration is read via one\n    (or max two, in the case of the presence of dataset-specific configuration)\n    calls to `git config`.  If this class is initialized with a Dataset\n    instance, it supports reading and writing configuration from\n    ``.datalad/config`` inside a dataset too. This file is committed to Git and\n    hence useful to ship certain configuration items with a dataset.\n\n    The API aims to provide the most significant read-access API of a\n    dictionary, the Python ConfigParser, and GitPython's config parser\n    implementations.\n\n    This class is presently not capable of efficiently writing multiple\n    configurations items at once.  Instead, each modification results in a\n    dedicated call to `git config`. This author thinks this is OK, as he\n    cannot think of a situation where a large number of items need to be\n    written during normal operation.\n\n    Each instance carries a public `overrides` attribute. This dictionary\n    contains variables that override any setting read from a file. The overrides\n    are persistent across reloads.\n\n    Any DATALAD_* environment variable is also presented as a configuration\n    item. Settings read from environment variables are not stored in any of the\n    configuration files, but are read dynamically from the environment at each\n    `reload()` call. Their values take precedence over any specification in\n    configuration files, and even overrides.\n\n    Parameters\n    ----------\n    dataset : Dataset, optional\n      If provided, all `git config` calls are executed in this dataset's\n      directory. Moreover, any modifications are, by default, directed to\n      this dataset's configuration file (which will be created on demand)\n    overrides : dict, optional\n      Variable overrides, see general class documentation for details.\n    source : {'any', 'local', 'dataset', 'dataset-local'}, optional\n      Which sources of configuration setting to consider. If 'dataset',\n      configuration items are only read from a dataset's persistent\n      configuration file, if any is present (the one in ``.datalad/config``, not\n      ``.git/config``); if 'local', any non-committed source is considered\n      (local and global configuration in Git config's terminology);\n      if 'dataset-local', persistent dataset configuration and local, but\n      not global or system configuration are considered; if 'any'\n      all possible sources of configuration are considered.\n    \"\"\"\n\n    _checked_git_identity = False\n\n    # Lock for running changing operation across multiple threads.\n    # Since config itself to the same path could\n    # potentially be created independently in multiple threads, and we might be\n    # modifying global config as well, making lock static should not allow more than\n    # one thread to  write at a time, even if to different repositories.\n    _run_lock = threading.Lock()\n\n    def __init__(self, dataset=None, overrides=None, source='any'):\n        if source not in ('any', 'local', 'dataset', 'dataset-local'):\n            raise ValueError(\n                'Unknown ConfigManager(source=) setting: {}'.format(source))\n        store = dict(\n            # store in a simple dict\n            # no subclassing, because we want to be largely read-only, and implement\n            # config writing separately\n            cfg={},\n            # track the files that jointly make up the config in this store\n            files=set(),\n            # and their modification times to be able to avoid needless unforced reloads\n            stats=None,\n        )\n        self._stores = dict(\n            # populated with info from git\n            git=store,\n            # only populated with info from commited dataset config\n            dataset=store.copy(),\n        )\n        # merged representation (the only one that existed pre datalad 0.14)\n        # will be built on initial reload\n        self._merged_store = {}\n\n        self._repo_dot_git = None\n        self._repo_pathobj = None\n        if dataset:\n            if hasattr(dataset, 'dot_git'):\n                self._repo_dot_git = dataset.dot_git\n                self._repo_pathobj = dataset.pathobj\n            elif dataset.repo:\n                self._repo_dot_git = dataset.repo.dot_git\n                self._repo_pathobj = dataset.repo.pathobj\n\n        self._config_cmd = ['git', 'config']\n        # public dict to store variables that always override any setting\n        # read from a file\n        # `hasattr()` is needed because `datalad.cfg` is generated upon first module\n        # import, hence when this code runs first, there cannot be any config manager\n        # to inherit from\n        self.overrides = datalad.cfg.overrides.copy() if hasattr(datalad, 'cfg') else {}\n        if overrides is not None:\n            self.overrides.update(overrides)\n        if dataset is None:\n            if source in ('dataset', 'dataset-local'):\n                raise ValueError(\n                    'ConfigManager configured to read dataset only, '\n                    'but no dataset given')\n            # The caller didn't specify a repository. Unset the git directory\n            # when calling 'git config' to prevent a repository in the current\n            # working directory from leaking configuration into the output.\n            self._config_cmd = ['git', '--git-dir=', 'config']\n\n        self._src_mode = source\n        run_kwargs = dict()\n        self._runner = None\n        if dataset is not None:\n            if hasattr(dataset, '_git_runner'):\n                self._runner = dataset._git_runner\n            elif dataset.repo:\n                self._runner = dataset.repo._git_runner\n            else:\n                # make sure we run the git config calls in the dataset\n                # to pick up the right config files\n                run_kwargs['cwd'] = dataset.path\n        if self._runner is None:\n            self._runner = GitWitlessRunner(**run_kwargs)\n\n        self.reload(force=True)\n\n        if not ConfigManager._checked_git_identity:\n            for cfg, envs in (\n                    ('user.name', ('GIT_AUTHOR_NAME', 'GIT_COMMITTER_NAME')),\n                    ('user.email', ('GIT_AUTHOR_EMAIL', 'GIT_COMMITTER_EMAIL'))):\n                if cfg not in self \\\n                        and not any(e in os.environ for e in envs):\n                    lgr.warning(\n                        \"It is highly recommended to configure Git before using \"\n                        \"DataLad. Set both 'user.name' and 'user.email' \"\n                        \"configuration variables.\"\n                    )\n            ConfigManager._checked_git_identity = True\n\n    def reload(self, force=False):\n        \"\"\"Reload all configuration items from the configured sources\n\n        If `force` is False, all files configuration was previously read from\n        are checked for differences in the modification times. If no difference\n        is found for any file no reload is performed. This mechanism will not\n        detect newly created global configuration files, use `force` in this case.\n        \"\"\"\n        run_args = ['-z', '-l', '--show-origin']\n\n        # update from desired config sources only\n        # 2-step strategy:\n        #   - load datalad dataset config from dataset\n        #   - load git config from all supported by git sources\n        # in doing so we always stay compatible with where Git gets its\n        # config from, but also allow to override persistent information\n        # from dataset locally or globally\n\n        # figure out what needs to be reloaded at all\n        to_run = {}\n        # committed dataset config\n        dataset_cfgfile = self._repo_pathobj / DATASET_CONFIG_FILE \\\n            if self._repo_pathobj else None\n        if (self._src_mode != 'local' and\n                dataset_cfgfile and\n                dataset_cfgfile.exists()) and (\n                force or self._need_reload(self._stores['dataset'])):\n            to_run['dataset'] = run_args + ['--file', str(dataset_cfgfile)]\n\n        if self._src_mode != 'dataset' and (\n                force or self._need_reload(self._stores['git'])):\n            to_run['git'] = run_args + ['--local'] \\\n                if self._src_mode == 'dataset-local' \\\n                else run_args\n\n        # reload everything that was found todo\n        while to_run:\n            store_id, runargs = to_run.popitem()\n            self._stores[store_id] = self._reload(runargs)\n\n        # always update the merged representation, even if we did not reload\n        # anything from a file. ENV or overrides could change independently\n        # start with the commit dataset config\n        merged = self._stores['dataset']['cfg'].copy()\n        # local config always takes precedence\n        merged.update(self._stores['git']['cfg'])\n        # superimpose overrides\n        merged.update(self.overrides)\n        # override with environment variables, unless we only want to read the\n        # dataset's commit config\n        if self._src_mode != 'dataset':\n            _update_from_env(merged)\n        self._merged_store = merged\n\n    def _need_reload(self, store):\n        storestats = store['stats']\n        if not storestats:\n            return True\n\n        # we have read files before\n        # check if any file we read from has changed\n        curstats = self._get_stats(store)\n        return any(curstats[f] != storestats[f] for f in store['files'])\n\n    def _reload(self, run_args):\n        # query git-config\n        stdout, stderr = self._run(\n            run_args,\n            protocol=StdOutErrCapture,\n            # always expect git-config to output utf-8\n            encoding='utf-8',\n        )\n        store = {}\n        store['cfg'], store['files'] = _parse_gitconfig_dump(\n            stdout, cwd=self._runner.cwd)\n\n        # update stats of config files, they have just been discovered\n        # and should still exist\n        store['stats'] = self._get_stats(store)\n        return store\n\n    @staticmethod\n    def _get_stats(store):\n        stats = {}\n        for f in store['files']:\n            if f.exists:\n                stat = f.stat()\n                stats[f] = _stat_result(stat.st_ino, stat.st_size, stat.st_ctime, stat.st_mtime)\n            else:\n                stats[f] = None\n        return stats\n\n    @_where_reload\n    def obtain(self, var, default=None, dialog_type=None, valtype=None,\n               store=False, where=None, reload=True, **kwargs):\n        \"\"\"\n        Convenience method to obtain settings interactively, if needed\n\n        A UI will be used to ask for user input in interactive sessions.\n        Questions to ask, and additional explanations can be passed directly\n        as arguments, or retrieved from a list of pre-configured items.\n\n        Additionally, this method allows for type conversion and storage\n        of obtained settings. Both aspects can also be pre-configured.\n\n        Parameters\n        ----------\n        var : str\n          Variable name including any section like `git config` expects them,\n          e.g. 'core.editor'\n        default : any type\n          In interactive sessions and if `store` is True, this default value\n          will be presented to the user for confirmation (or modification).\n          In all other cases, this value will be silently assigned unless\n          there is an existing configuration setting.\n        dialog_type : {'question', 'yesno', None}\n          Which dialog type to use in interactive sessions. If `None`,\n          pre-configured UI options are used.\n        store : bool\n          Whether to store the obtained value (or default)\n        %s\n        `**kwargs`\n          Additional arguments for the UI function call, such as a question\n          `text`.\n        \"\"\"\n        # do local import, as this module is import prominently and the\n        # could theroetically import all kind of weired things for type\n        # conversion\n        from datalad.interface.common_cfg import definitions as cfg_defs\n        # fetch what we know about this variable\n        cdef = cfg_defs.get(var, {})\n        # type conversion setup\n        if valtype is None and 'type' in cdef:\n            valtype = cdef['type']\n        if valtype is None:\n            valtype = lambda x: x\n\n        # any default?\n        if default is None and 'default' in cdef:\n            default = cdef['default']\n\n        _value = None\n        if var in self:\n            # nothing needs to be obtained, it is all here already\n            _value = self[var]\n        elif store is False and default is not None:\n            # nothing will be stored, and we have a default -> no user confirmation\n            # we cannot use logging, because we want to use the config to confiugre\n            # the logging\n            #lgr.debug('using default {} for config setting {}'.format(default, var))\n            _value = default\n\n        if _value is not None:\n            # we got everything we need and can exit early\n            try:\n                return valtype(_value)\n            except Exception as e:\n                raise ValueError(\n                    \"value '{}' of existing configuration for '{}' cannot be \"\n                    \"converted to the desired type '{}' ({})\".format(\n                        _value, var, valtype, exc_str(e)))\n\n        # now we need to try to obtain something from the user\n        from datalad.ui import ui\n\n        # configure UI\n        dialog_opts = kwargs\n        if dialog_type is None:  # no override\n            # check for common knowledge on how to obtain a value\n            if 'ui' in cdef:\n                dialog_type = cdef['ui'][0]\n                # pull standard dialog settings\n                dialog_opts = cdef['ui'][1]\n                # update with input\n                dialog_opts.update(kwargs)\n\n        if (not ui.is_interactive or dialog_type is None) and default is None:\n            raise RuntimeError(\n                \"cannot obtain value for configuration item '{}', \"\n                \"not preconfigured, no default, no UI available\".format(var))\n\n        if not hasattr(ui, dialog_type):\n            raise ValueError(\"UI '{}' does not support dialog type '{}'\".format(\n                ui, dialog_type))\n\n        # configure storage destination, if needed\n        if store:\n            if where is None and 'destination' in cdef:\n                where = cdef['destination']\n            if where is None:\n                raise ValueError(\n                    \"request to store configuration item '{}', but no \"\n                    \"storage destination specified\".format(var))\n\n        # obtain via UI\n        dialog = getattr(ui, dialog_type)\n        _value = dialog(default=default, **dialog_opts)\n\n        if _value is None:\n            # we got nothing\n            if default is None:\n                raise RuntimeError(\n                    \"could not obtain value for configuration item '{}', \"\n                    \"not preconfigured, no default\".format(var))\n            # XXX maybe we should return default here, even it was returned\n            # from the UI -- if that is even possible\n\n        # execute type conversion before storing to check that we got\n        # something that looks like what we want\n        try:\n            value = valtype(_value)\n        except Exception as e:\n            raise ValueError(\n                \"cannot convert user input `{}` to desired type ({})\".format(\n                    _value, exc_str(e)))\n            # XXX we could consider \"looping\" until we have a value of proper\n            # type in case of a user typo...\n\n        if store:\n            # store value as it was before any conversion, needs to be str\n            # anyway\n            # needs string conversion nevertheless, because default could come\n            # in as something else\n            self.add(var, '{}'.format(_value), where=where, reload=reload)\n        return value\n\n    def __repr__(self):\n        # give full list of all tracked config files, plus overrides\n        return \"ConfigManager({}{})\".format(\n            [str(p) for p in self._stores['dataset']['files'].union(\n                self._stores['git']['files'])],\n            ', overrides={!r}'.format(self.overrides) if self.overrides else '',\n        )\n\n    def __str__(self):\n        # give path of dataset, if there is any, plus overrides\n        return \"ConfigManager({}{})\".format(\n            self._repo_pathobj if self._repo_pathobj else '',\n            'with overrides' if self.overrides else '',\n        )\n\n    #\n    # Compatibility with dict API\n    #\n    def __len__(self):\n        return len(self._merged_store)\n\n    def __getitem__(self, key):\n        return self._merged_store.__getitem__(key)\n\n    def __contains__(self, key):\n        return self._merged_store.__contains__(key)\n\n    def keys(self):\n        \"\"\"Returns list of configuration item names\"\"\"\n        return self._merged_store.keys()\n\n    # XXX should this be *args?\n    def get(self, key, default=None, get_all=False):\n        \"\"\"D.get(k[,d]) -> D[k] if k in D, else d.  d defaults to None.\n\n        Parameters\n        ----------\n        default : optional\n          Value to return when key is not present. `None` by default.\n        get_all : bool, optional\n          If True, return all values of multiple identical configuration keys.\n          By default only the last specified value is returned.\n        \"\"\"\n        try:\n            val = self[key]\n            if get_all or not isinstance(val, tuple):\n                return val\n            else:\n                return val[-1]\n        except KeyError:\n            # return as-is, default could be a tuple, hence do not subject to\n            # get_all processing\n            return default\n\n    def get_from_source(self, source, key, default=None):\n        \"\"\"Like get(), but a source can be specific.\n\n        If `source` is 'dataset', only the commited configuration is queried,\n        overrides are applied. In the case of 'local', the committed\n        configuration is ignored, but overrides and configuration from\n        environment variables are applied as usual.\n        \"\"\"\n        if source not in ('dataset', 'local'):\n            raise ValueError(\"source must be 'dataset' or 'local'\")\n        if source == 'dataset':\n            return self.overrides.get(\n                key,\n                self._stores['dataset']['cfg'].get(\n                    key,\n                    default))\n        else:\n            if key not in self._stores['dataset']['cfg']:\n                # the key is not in the committed config, hence we can\n                # just report based on the merged representation\n                return self.get(key, default)\n            else:\n                # expensive case, rebuild a config without the committed\n                # dataset config contributing\n                env = {}\n                _update_from_env(env)\n                return env.get(\n                    key,\n                    self.overrides.get(\n                        key,\n                        self._stores['local']['cfg'].get(\n                            key,\n                            default)))\n\n    #\n    # Compatibility with ConfigParser API\n    #\n    def sections(self):\n        \"\"\"Returns a list of the sections available\"\"\"\n        return list(set([cfg_section_regex.match(k).group(1) for k in self._merged_store]))\n\n    def options(self, section):\n        \"\"\"Returns a list of options available in the specified section.\"\"\"\n        opts = []\n        for k in self._merged_store:\n            sec, opt = cfg_sectionoption_regex.match(k).groups()\n            if sec == section:\n                opts.append(opt)\n        return opts\n\n    def has_section(self, section):\n        \"\"\"Indicates whether a section is present in the configuration\"\"\"\n        for k in self._merged_store:\n            if k.startswith(section):\n                return True\n        return False\n\n    def has_option(self, section, option):\n        \"\"\"If the given section exists, and contains the given option\"\"\"\n        for k in self._merged_store:\n            sec, opt = cfg_sectionoption_regex.match(k).groups()\n            if sec == section and opt == option:\n                return True\n        return False\n\n    def _get_type(self, typefn, section, option):\n        key = '.'.join([section, option])\n        # Mimic the handling of get_value(..., default=None), while still going\n        # through get() in order to get its default tuple handling.\n        if key not in self:\n            raise KeyError(key)\n        return typefn(self.get(key))\n\n    def getint(self, section, option):\n        \"\"\"A convenience method which coerces the option value to an integer\"\"\"\n        return self._get_type(int, section, option)\n\n    def getfloat(self, section, option):\n        \"\"\"A convenience method which coerces the option value to a float\"\"\"\n        return self._get_type(float, section, option)\n\n    def getbool(self, section, option, default=None):\n        \"\"\"A convenience method which coerces the option value to a bool\n\n        Values \"on\", \"yes\", \"true\" and any int!=0 are considered True\n        Values which evaluate to bool False, \"off\", \"no\", \"false\" are considered\n        False\n        TypeError is raised for other values.\n        \"\"\"\n        key = '.'.join([section, option])\n        # Mimic the handling of get_value(..., default=None), while still going\n        # through get() in order to get its default tuple handling.\n        if default is None and key not in self:\n            raise KeyError(key)\n        val = self.get(key, default=default)\n        if val is None:  # no value at all, git treats it as True\n            return True\n        return anything2bool(val)\n\n    # this is a hybrid of ConfigParser and dict API\n    def items(self, section=None):\n        \"\"\"Return a list of (name, value) pairs for each option\n\n        Optionally limited to a given section.\n        \"\"\"\n        if section is None:\n            return self._merged_store.items()\n        return [(k, v) for k, v in self._merged_store.items()\n                if cfg_section_regex.match(k).group(1) == section]\n\n    #\n    # Compatibility with GitPython's ConfigParser\n    #\n    def get_value(self, section, option, default=None):\n        \"\"\"Like `get()`, but with an optional default value\n\n        If the default is not None, the given default value will be returned in\n        case the option did not exist. This behavior imitates GitPython's\n        config parser.\n        \"\"\"\n        try:\n            return self['.'.join((section, option))]\n        except KeyError as e:\n            # this strange dance is needed because gitpython does it this way\n            if default is not None:\n                return default\n            else:\n                raise e\n\n    #\n    # Modify configuration (proxy respective git-config call)\n    #\n    @_where_reload\n    def _run(self, args, where=None, reload=False, **kwargs):\n        \"\"\"Centralized helper to run \"git config\" calls\n\n        Parameters\n        ----------\n        args : list\n          Arguments to pass for git config\n        %s\n        **kwargs\n          Keywords arguments for Runner's call\n        \"\"\"\n        if where:\n            args = self._get_location_args(where) + args\n        if '-l' in args:\n            # we are just reading, no need to reload, no need to lock\n            out = self._runner.run(self._config_cmd + args, **kwargs)\n            return out['stdout'], out['stderr']\n\n        # all other calls are modifications\n        if '--file' in args:\n            # all paths we are passing are absolute\n            custom_file = Path(args[args.index('--file') + 1])\n            custom_file.parent.mkdir(exist_ok=True)\n        lockfile = None\n        if self._repo_dot_git and ('--local' in args or '--file' in args):\n            # modification of config in a dataset\n            lockfile = self._repo_dot_git / 'config.dataladlock'\n        else:\n            # follow pattern in downloaders for lockfile location\n            lockfile = Path(self.obtain('datalad.locations.cache')) \\\n                / 'locks' / 'gitconfig.lck'\n\n        with ConfigManager._run_lock, InterProcessLock(lockfile, logger=lgr):\n            out = self._runner.run(self._config_cmd + args, **kwargs)\n\n        if reload:\n            self.reload()\n        return out['stdout'], out['stderr']\n\n    def _get_location_args(self, where, args=None):\n        if args is None:\n            args = []\n        cfg_labels = ('dataset', 'local', 'global', 'override')\n        if where not in cfg_labels:\n            raise ValueError(\n                \"unknown configuration label '{}' (not in {})\".format(\n                    where, cfg_labels))\n        if where == 'dataset':\n            if not self._repo_pathobj:\n                raise ValueError(\n                    'ConfigManager cannot store configuration to dataset, '\n                    'none specified')\n            dataset_cfgfile = self._repo_pathobj / DATASET_CONFIG_FILE\n            args.extend(['--file', str(dataset_cfgfile)])\n        elif where == 'global':\n            args.append('--global')\n        elif where == 'local':\n            args.append('--local')\n        return args\n\n    @_where_reload\n    def add(self, var, value, where='dataset', reload=True):\n        \"\"\"Add a configuration variable and value\n\n        Parameters\n        ----------\n        var : str\n          Variable name including any section like `git config` expects them, e.g.\n          'core.editor'\n        value : str\n          Variable value\n        %s\"\"\"\n        if where == 'override':\n            from datalad.utils import ensure_list\n            val = ensure_list(self.overrides.pop(var, None))\n            val.append(value)\n            self.overrides[var] = val[0] if len(val) == 1 else val\n            if reload:\n                self.reload(force=True)\n            return\n\n        self._run(['--add', var, value], where=where, reload=reload,\n                  protocol=StdOutErrCapture)\n\n    @_where_reload\n    def set(self, var, value, where='dataset', reload=True, force=False):\n        \"\"\"Set a variable to a value.\n\n        In opposition to `add`, this replaces the value of `var` if there is\n        one already.\n\n        Parameters\n        ----------\n        var : str\n          Variable name including any section like `git config` expects them, e.g.\n          'core.editor'\n        value : str\n          Variable value\n        force: bool\n          if set, replaces all occurrences of `var` by a single one with the\n          given `value`. Otherwise raise if multiple entries for `var` exist\n          already\n        %s\"\"\"\n        if where == 'override':\n            self.overrides[var] = value\n            if reload:\n                self.reload(force=True)\n            return\n\n        from datalad.support.gitrepo import to_options\n\n        self._run(to_options(replace_all=force) + [var, value],\n                  where=where, reload=reload, protocol=StdOutErrCapture)\n\n    @_where_reload\n    def rename_section(self, old, new, where='dataset', reload=True):\n        \"\"\"Rename a configuration section\n\n        Parameters\n        ----------\n        old : str\n          Name of the section to rename.\n        new : str\n          Name of the section to rename to.\n        %s\"\"\"\n        if where == 'override':\n            self.overrides = {\n                (new + k[len(old):]) if k.startswith(old + '.') else k: v\n                for k, v in self.overrides.items()\n            }\n            if reload:\n                self.reload(force=True)\n            return\n\n        self._run(['--rename-section', old, new], where=where, reload=reload)\n\n    @_where_reload\n    def remove_section(self, sec, where='dataset', reload=True):\n        \"\"\"Rename a configuration section\n\n        Parameters\n        ----------\n        sec : str\n          Name of the section to remove.\n        %s\"\"\"\n        if where == 'override':\n            self.overrides = {\n                k: v\n                for k, v in self.overrides.items()\n                if not k.startswith(sec + '.')\n            }\n            if reload:\n                self.reload(force=True)\n            return\n\n        self._run(['--remove-section', sec], where=where, reload=reload)\n\n    @_where_reload\n    def unset(self, var, where='dataset', reload=True):\n        \"\"\"Remove all occurrences of a variable\n\n        Parameters\n        ----------\n        var : str\n          Name of the variable to remove\n        %s\"\"\"\n        if where == 'override':\n            self.overrides.pop(var, None)\n            if reload:\n                self.reload(force=True)\n            return\n\n        # use unset all as it is simpler for now\n        self._run(['--unset-all', var], where=where, reload=reload)\n\n\ndef rewrite_url(cfg, url):\n    \"\"\"Any matching 'url.<base>.insteadOf' configuration is applied\n\n    Any URL that starts with such a configuration will be rewritten\n    to start, instead, with <base>. When more than one insteadOf\n    strings match a given URL, the longest match is used.\n\n    Parameters\n    ----------\n    cfg : ConfigManager or dict\n      dict-like with configuration variable name/value-pairs.\n    url : str\n      URL to be rewritten, if matching configuration is found.\n\n    Returns\n    -------\n    str\n      Rewritten or unmodified URL.\n    \"\"\"\n    insteadof = {\n        # only leave the base url\n        k[4:-10]: v\n        for k, v in cfg.items()\n        if k.startswith('url.') and k.endswith('.insteadof')\n    }\n\n    # all config that applies\n    matches = {\n        key: v\n        for key, val in insteadof.items()\n        for v in (val if isinstance(val, tuple) else (val,))\n        if url.startswith(v)\n    }\n    # find longest match, like Git does\n    if matches:\n        rewrite_base, match = sorted(\n            matches.items(),\n            key=lambda x: len(x[1]),\n            reverse=True,\n        )[0]\n        if sum(match == v for v in matches.values()) > 1:\n            lgr.warning(\n                \"Ignoring URL rewrite configuration for '%s', \"\n                \"multiple conflicting definitions exists: %s\",\n                match,\n                ['url.{}.insteadof'.format(k)\n                 for k, v in matches.items()\n                 if v == match]\n            )\n        else:\n            url = '{}{}'.format(rewrite_base, url[len(match):])\n    return url\n\n\n# for convenience, bind to class too\nConfigManager.rewrite_url = rewrite_url\n\n#\n# Helpers for bypassing git-config when _writing_ config items,\n# mostly useful when a large number of changes needs to be made\n# and directly file manipulation without a safety net is worth\n# the risk for performance reasons.\n#\n\ndef quote_config(v):\n    \"\"\"Helper to perform minimal quoting of config keys/value parts\n\n    Parameters\n    ----------\n    v : str\n      To-be-quoted string\n    \"\"\"\n    white = (' ', '\\t')\n    # backslashes need to be quoted in any case\n    v = v.replace('\\\\', '\\\\\\\\')\n    # must not have additional unquoted quotes\n    v = v.replace('\"', '\\\\\"')\n    if v[0] in white or v[-1] in white:\n        # quoting the value due to leading/trailing whitespace\n        v = '\"{}\"'.format(v)\n    return v\n\n\ndef write_config_section(fobj, suite, name, props):\n    \"\"\"Write a config section with (multiple) settings.\n\n    Parameters\n    ----------\n    fobj : File\n       Opened target file\n    suite : str\n       First item of the section name, e.g. 'submodule', or\n       'datalad'\n    name : str\n       Remainder of the section name\n    props : dict\n       Keys are configuration setting names within the section\n       context (i.e. not duplicating `suite` and/or `name`, values\n       are configuration setting values.\n    \"\"\"\n    fmt = '[{_suite_} {_q_}{_name_}{_q_}]\\n'\n    for p in props:\n        fmt += '\\t{p} = {{{p}}}\\n'.format(p=p)\n    quoted_name = quote_config(name)\n    fobj.write(\n        fmt.format(\n            _suite_=suite,\n            _q_='' if quoted_name.startswith('\"') else '\"',\n            _name_=quoted_name,\n            **{k: quote_config(v) for k, v in props.items()}))\n",
            "new_file_content": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"\n\"\"\"\n\nfrom collections import namedtuple\n\nimport json\nimport threading\nfrom fasteners import InterProcessLock\nfrom functools import lru_cache\nimport datalad\nfrom datalad.consts import (\n    DATASET_CONFIG_FILE,\n)\nfrom datalad.cmd import (\n    GitWitlessRunner,\n    StdOutErrCapture,\n)\nfrom datalad.dochelpers import exc_str\n\nimport re\nimport os\nfrom pathlib import Path\n\nimport logging\nlgr = logging.getLogger('datalad.config')\n\ncfg_kv_regex = re.compile(r'(^.*)\\n(.*)$', flags=re.MULTILINE)\ncfg_section_regex = re.compile(r'(.*)\\.[^.]+')\ncfg_sectionoption_regex = re.compile(r'(.*)\\.([^.]+)')\n\n\n_where_reload_doc = \"\"\"\n        where : {'dataset', 'local', 'global', 'override'}, optional\n          Indicator which configuration file to modify. 'dataset' indicates the\n          persistent configuration in .datalad/config of a dataset; 'local'\n          the configuration of a dataset's Git repository in .git/config;\n          'global' refers to the general configuration that is not specific to\n          a single repository (usually in $USER/.gitconfig); 'override'\n          limits the modification to the ConfigManager instance, and the\n          assigned value overrides any setting from any other source.\n        reload : bool\n          Flag whether to reload the configuration from file(s) after\n          modification. This can be disable to make multiple sequential\n          modifications slightly more efficient.\"\"\".lstrip()\n\n# Selection of os.stat_result fields we care to collect/compare to judge\n# on either file has changed to warrant reload of configuration.\n_stat_result = namedtuple('_stat_result', 'st_ino st_size st_ctime st_mtime')\n\n\n# we cannot import external_versions here, as the cfg comes before anything\n# and we would have circular imports\n@lru_cache()\ndef get_git_version(runner=None):\n    \"\"\"Return version of available git\"\"\"\n    runner = runner or GitWitlessRunner()\n    return runner.run('git version'.split(),\n                      protocol=StdOutErrCapture)['stdout'].split()[2]\n\n\ndef _where_reload(obj):\n    \"\"\"Helper decorator to simplify providing repetitive docstring\"\"\"\n    obj.__doc__ = obj.__doc__ % _where_reload_doc\n    return obj\n\n\n# TODO document and make \"public\" (used in GitRepo too)\ndef _parse_gitconfig_dump(dump, cwd=None, multi_value=True):\n    \"\"\"Parse a dump-string from `git config -z --list`\n\n    Parameters\n    ----------\n    dump : str\n      Null-byte separated output\n    cwd : path-like, optional\n      Use this path to convert relative paths for origin reports\n      into absolute paths\n    multi_value : bool, optional\n      If True, report values from multiple specifications of the\n      same key as a tuple of values assigned to this key. Otherwise,\n      the last configuration is reported.\n    \"\"\"\n    dct = {}\n    fileset = set()\n    for line in dump.split('\\0'):\n        if not line:\n            continue\n        if line.startswith('file:'):\n            # origin line\n            fname = Path(line[5:])\n            if not fname.is_absolute():\n                fname = Path(cwd) / fname if cwd else Path.cwd() / fname\n            fileset.add(fname)\n            continue\n        if line.startswith('command line:'):\n            # nothing we could handle\n            continue\n        kv_match = cfg_kv_regex.match(line)\n        if kv_match:\n            k, v = kv_match.groups()\n        else:\n            # could be just a key without = value, which git treats as True\n            # if asked for a bool\n            k, v = line, None\n        present_v = dct.get(k, None)\n        if present_v is None or not multi_value:\n            dct[k] = v\n        else:\n            if isinstance(present_v, tuple):\n                dct[k] = present_v + (v,)\n            else:\n                dct[k] = (present_v, v)\n    return dct, fileset\n\n\ndef _update_from_env(store):\n    overrides = {}\n    dct = {}\n    for k in os.environ:\n        if k == \"DATALAD_CONFIG_OVERRIDES_JSON\":\n            try:\n                overrides = json.loads(os.environ[k])\n            except json.decoder.JSONDecodeError as exc:\n                lgr.warning(\"Failed to load DATALAD_CONFIG_OVERRIDES_JSON: %s\",\n                            exc)\n        elif k.startswith('DATALAD_'):\n            dct[k.replace('__', '-').replace('_', '.').lower()] = os.environ[k]\n    if overrides:\n        store.update(overrides)\n    store.update(dct)\n\n\ndef anything2bool(val):\n    if hasattr(val, 'lower'):\n        val = val.lower()\n    if val in {\"off\", \"no\", \"false\", \"0\"} or not bool(val):\n        return False\n    elif val in {\"on\", \"yes\", \"true\", True} \\\n            or (hasattr(val, 'isdigit') and val.isdigit() and int(val)) \\\n            or isinstance(val, int) and val:\n        return True\n    else:\n        raise TypeError(\n            \"Got value %s which could not be interpreted as a boolean\"\n            % repr(val))\n\n\nclass ConfigManager(object):\n    \"\"\"Thin wrapper around `git-config` with support for a dataset configuration.\n\n    The general idea is to have an object that is primarily used to read/query\n    configuration option.  Upon creation, current configuration is read via one\n    (or max two, in the case of the presence of dataset-specific configuration)\n    calls to `git config`.  If this class is initialized with a Dataset\n    instance, it supports reading and writing configuration from\n    ``.datalad/config`` inside a dataset too. This file is committed to Git and\n    hence useful to ship certain configuration items with a dataset.\n\n    The API aims to provide the most significant read-access API of a\n    dictionary, the Python ConfigParser, and GitPython's config parser\n    implementations.\n\n    This class is presently not capable of efficiently writing multiple\n    configurations items at once.  Instead, each modification results in a\n    dedicated call to `git config`. This author thinks this is OK, as he\n    cannot think of a situation where a large number of items need to be\n    written during normal operation.\n\n    Each instance carries a public `overrides` attribute. This dictionary\n    contains variables that override any setting read from a file. The overrides\n    are persistent across reloads.\n\n    Any DATALAD_* environment variable is also presented as a configuration\n    item. Settings read from environment variables are not stored in any of the\n    configuration files, but are read dynamically from the environment at each\n    `reload()` call. Their values take precedence over any specification in\n    configuration files, and even overrides.\n\n    Parameters\n    ----------\n    dataset : Dataset, optional\n      If provided, all `git config` calls are executed in this dataset's\n      directory. Moreover, any modifications are, by default, directed to\n      this dataset's configuration file (which will be created on demand)\n    overrides : dict, optional\n      Variable overrides, see general class documentation for details.\n    source : {'any', 'local', 'dataset', 'dataset-local'}, optional\n      Which sources of configuration setting to consider. If 'dataset',\n      configuration items are only read from a dataset's persistent\n      configuration file, if any is present (the one in ``.datalad/config``, not\n      ``.git/config``); if 'local', any non-committed source is considered\n      (local and global configuration in Git config's terminology);\n      if 'dataset-local', persistent dataset configuration and local, but\n      not global or system configuration are considered; if 'any'\n      all possible sources of configuration are considered.\n    \"\"\"\n\n    _checked_git_identity = False\n\n    # Lock for running changing operation across multiple threads.\n    # Since config itself to the same path could\n    # potentially be created independently in multiple threads, and we might be\n    # modifying global config as well, making lock static should not allow more than\n    # one thread to  write at a time, even if to different repositories.\n    _run_lock = threading.Lock()\n\n    def __init__(self, dataset=None, overrides=None, source='any'):\n        if source not in ('any', 'local', 'dataset', 'dataset-local'):\n            raise ValueError(\n                'Unknown ConfigManager(source=) setting: {}'.format(source))\n        store = dict(\n            # store in a simple dict\n            # no subclassing, because we want to be largely read-only, and implement\n            # config writing separately\n            cfg={},\n            # track the files that jointly make up the config in this store\n            files=set(),\n            # and their modification times to be able to avoid needless unforced reloads\n            stats=None,\n        )\n        self._stores = dict(\n            # populated with info from git\n            git=store,\n            # only populated with info from commited dataset config\n            dataset=store.copy(),\n        )\n        # merged representation (the only one that existed pre datalad 0.14)\n        # will be built on initial reload\n        self._merged_store = {}\n\n        self._repo_dot_git = None\n        self._repo_pathobj = None\n        if dataset:\n            if hasattr(dataset, 'dot_git'):\n                self._repo_dot_git = dataset.dot_git\n                self._repo_pathobj = dataset.pathobj\n            elif dataset.repo:\n                self._repo_dot_git = dataset.repo.dot_git\n                self._repo_pathobj = dataset.repo.pathobj\n\n        self._config_cmd = ['git', 'config']\n        # public dict to store variables that always override any setting\n        # read from a file\n        # `hasattr()` is needed because `datalad.cfg` is generated upon first module\n        # import, hence when this code runs first, there cannot be any config manager\n        # to inherit from\n        self.overrides = datalad.cfg.overrides.copy() if hasattr(datalad, 'cfg') else {}\n        if overrides is not None:\n            self.overrides.update(overrides)\n        if dataset is None:\n            if source in ('dataset', 'dataset-local'):\n                raise ValueError(\n                    'ConfigManager configured to read dataset only, '\n                    'but no dataset given')\n            # The caller didn't specify a repository. Unset the git directory\n            # when calling 'git config' to prevent a repository in the current\n            # working directory from leaking configuration into the output.\n            self._config_cmd = ['git', '--git-dir=', 'config']\n\n        self._src_mode = source\n        run_kwargs = dict()\n        self._runner = None\n        if dataset is not None:\n            if hasattr(dataset, '_git_runner'):\n                self._runner = dataset._git_runner\n            elif dataset.repo:\n                self._runner = dataset.repo._git_runner\n            else:\n                # make sure we run the git config calls in the dataset\n                # to pick up the right config files\n                run_kwargs['cwd'] = dataset.path\n        if self._runner is None:\n            self._runner = GitWitlessRunner(**run_kwargs)\n\n        self.reload(force=True)\n\n        if not ConfigManager._checked_git_identity:\n            for cfg, envs in (\n                    ('user.name', ('GIT_AUTHOR_NAME', 'GIT_COMMITTER_NAME')),\n                    ('user.email', ('GIT_AUTHOR_EMAIL', 'GIT_COMMITTER_EMAIL'))):\n                if cfg not in self \\\n                        and not any(e in os.environ for e in envs):\n                    lgr.warning(\n                        \"It is highly recommended to configure Git before using \"\n                        \"DataLad. Set both 'user.name' and 'user.email' \"\n                        \"configuration variables.\"\n                    )\n            ConfigManager._checked_git_identity = True\n\n    def reload(self, force=False):\n        \"\"\"Reload all configuration items from the configured sources\n\n        If `force` is False, all files configuration was previously read from\n        are checked for differences in the modification times. If no difference\n        is found for any file no reload is performed. This mechanism will not\n        detect newly created global configuration files, use `force` in this case.\n        \"\"\"\n        run_args = ['-z', '-l', '--show-origin']\n\n        # update from desired config sources only\n        # 2-step strategy:\n        #   - load datalad dataset config from dataset\n        #   - load git config from all supported by git sources\n        # in doing so we always stay compatible with where Git gets its\n        # config from, but also allow to override persistent information\n        # from dataset locally or globally\n\n        # figure out what needs to be reloaded at all\n        to_run = {}\n        # committed dataset config\n        dataset_cfgfile = self._repo_pathobj / DATASET_CONFIG_FILE \\\n            if self._repo_pathobj else None\n        if (self._src_mode != 'local' and\n                dataset_cfgfile and\n                dataset_cfgfile.exists()) and (\n                force or self._need_reload(self._stores['dataset'])):\n            to_run['dataset'] = run_args + ['--file', str(dataset_cfgfile)]\n\n        if self._src_mode != 'dataset' and (\n                force or self._need_reload(self._stores['git'])):\n            to_run['git'] = run_args + ['--local'] \\\n                if self._src_mode == 'dataset-local' \\\n                else run_args\n\n        # reload everything that was found todo\n        while to_run:\n            store_id, runargs = to_run.popitem()\n            self._stores[store_id] = self._reload(runargs)\n\n        # always update the merged representation, even if we did not reload\n        # anything from a file. ENV or overrides could change independently\n        # start with the commit dataset config\n        merged = self._stores['dataset']['cfg'].copy()\n        # local config always takes precedence\n        merged.update(self._stores['git']['cfg'])\n        # superimpose overrides\n        merged.update(self.overrides)\n        # override with environment variables, unless we only want to read the\n        # dataset's commit config\n        if self._src_mode != 'dataset':\n            _update_from_env(merged)\n        self._merged_store = merged\n\n    def _need_reload(self, store):\n        storestats = store['stats']\n        if not storestats:\n            return True\n\n        # we have read files before\n        # check if any file we read from has changed\n        curstats = self._get_stats(store)\n        return any(curstats[f] != storestats[f] for f in store['files'])\n\n    def _reload(self, run_args):\n        # query git-config\n        stdout, stderr = self._run(\n            run_args,\n            protocol=StdOutErrCapture,\n            # always expect git-config to output utf-8\n            encoding='utf-8',\n        )\n        store = {}\n        store['cfg'], store['files'] = _parse_gitconfig_dump(\n            stdout, cwd=self._runner.cwd)\n\n        # update stats of config files, they have just been discovered\n        # and should still exist\n        store['stats'] = self._get_stats(store)\n        return store\n\n    @staticmethod\n    def _get_stats(store):\n        stats = {}\n        for f in store['files']:\n            if f.exists:\n                stat = f.stat()\n                stats[f] = _stat_result(stat.st_ino, stat.st_size, stat.st_ctime, stat.st_mtime)\n            else:\n                stats[f] = None\n        return stats\n\n    @_where_reload\n    def obtain(self, var, default=None, dialog_type=None, valtype=None,\n               store=False, where=None, reload=True, **kwargs):\n        \"\"\"\n        Convenience method to obtain settings interactively, if needed\n\n        A UI will be used to ask for user input in interactive sessions.\n        Questions to ask, and additional explanations can be passed directly\n        as arguments, or retrieved from a list of pre-configured items.\n\n        Additionally, this method allows for type conversion and storage\n        of obtained settings. Both aspects can also be pre-configured.\n\n        Parameters\n        ----------\n        var : str\n          Variable name including any section like `git config` expects them,\n          e.g. 'core.editor'\n        default : any type\n          In interactive sessions and if `store` is True, this default value\n          will be presented to the user for confirmation (or modification).\n          In all other cases, this value will be silently assigned unless\n          there is an existing configuration setting.\n        dialog_type : {'question', 'yesno', None}\n          Which dialog type to use in interactive sessions. If `None`,\n          pre-configured UI options are used.\n        store : bool\n          Whether to store the obtained value (or default)\n        %s\n        `**kwargs`\n          Additional arguments for the UI function call, such as a question\n          `text`.\n        \"\"\"\n        # do local import, as this module is import prominently and the\n        # could theroetically import all kind of weired things for type\n        # conversion\n        from datalad.interface.common_cfg import definitions as cfg_defs\n        # fetch what we know about this variable\n        cdef = cfg_defs.get(var, {})\n        # type conversion setup\n        if valtype is None and 'type' in cdef:\n            valtype = cdef['type']\n        if valtype is None:\n            valtype = lambda x: x\n\n        # any default?\n        if default is None and 'default' in cdef:\n            default = cdef['default']\n\n        _value = None\n        if var in self:\n            # nothing needs to be obtained, it is all here already\n            _value = self[var]\n        elif store is False and default is not None:\n            # nothing will be stored, and we have a default -> no user confirmation\n            # we cannot use logging, because we want to use the config to confiugre\n            # the logging\n            #lgr.debug('using default {} for config setting {}'.format(default, var))\n            _value = default\n\n        if _value is not None:\n            # we got everything we need and can exit early\n            try:\n                return valtype(_value)\n            except Exception as e:\n                raise ValueError(\n                    \"value '{}' of existing configuration for '{}' cannot be \"\n                    \"converted to the desired type '{}' ({})\".format(\n                        _value, var, valtype, exc_str(e)))\n\n        # now we need to try to obtain something from the user\n        from datalad.ui import ui\n\n        # configure UI\n        dialog_opts = kwargs\n        if dialog_type is None:  # no override\n            # check for common knowledge on how to obtain a value\n            if 'ui' in cdef:\n                dialog_type = cdef['ui'][0]\n                # pull standard dialog settings\n                dialog_opts = cdef['ui'][1]\n                # update with input\n                dialog_opts.update(kwargs)\n\n        if (not ui.is_interactive or dialog_type is None) and default is None:\n            raise RuntimeError(\n                \"cannot obtain value for configuration item '{}', \"\n                \"not preconfigured, no default, no UI available\".format(var))\n\n        if not hasattr(ui, dialog_type):\n            raise ValueError(\"UI '{}' does not support dialog type '{}'\".format(\n                ui, dialog_type))\n\n        # configure storage destination, if needed\n        if store:\n            if where is None and 'destination' in cdef:\n                where = cdef['destination']\n            if where is None:\n                raise ValueError(\n                    \"request to store configuration item '{}', but no \"\n                    \"storage destination specified\".format(var))\n\n        # obtain via UI\n        dialog = getattr(ui, dialog_type)\n        _value = dialog(default=default, **dialog_opts)\n\n        if _value is None:\n            # we got nothing\n            if default is None:\n                raise RuntimeError(\n                    \"could not obtain value for configuration item '{}', \"\n                    \"not preconfigured, no default\".format(var))\n            # XXX maybe we should return default here, even it was returned\n            # from the UI -- if that is even possible\n\n        # execute type conversion before storing to check that we got\n        # something that looks like what we want\n        try:\n            value = valtype(_value)\n        except Exception as e:\n            raise ValueError(\n                \"cannot convert user input `{}` to desired type ({})\".format(\n                    _value, exc_str(e)))\n            # XXX we could consider \"looping\" until we have a value of proper\n            # type in case of a user typo...\n\n        if store:\n            # store value as it was before any conversion, needs to be str\n            # anyway\n            # needs string conversion nevertheless, because default could come\n            # in as something else\n            self.add(var, '{}'.format(_value), where=where, reload=reload)\n        return value\n\n    def __repr__(self):\n        # give full list of all tracked config files, plus overrides\n        return \"ConfigManager({}{})\".format(\n            [str(p) for p in self._stores['dataset']['files'].union(\n                self._stores['git']['files'])],\n            ', overrides={!r}'.format(self.overrides) if self.overrides else '',\n        )\n\n    def __str__(self):\n        # give path of dataset, if there is any, plus overrides\n        return \"ConfigManager({}{})\".format(\n            self._repo_pathobj if self._repo_pathobj else '',\n            'with overrides' if self.overrides else '',\n        )\n\n    #\n    # Compatibility with dict API\n    #\n    def __len__(self):\n        return len(self._merged_store)\n\n    def __getitem__(self, key):\n        return self._merged_store.__getitem__(key)\n\n    def __contains__(self, key):\n        return self._merged_store.__contains__(key)\n\n    def keys(self):\n        \"\"\"Returns list of configuration item names\"\"\"\n        return self._merged_store.keys()\n\n    # XXX should this be *args?\n    def get(self, key, default=None, get_all=False):\n        \"\"\"D.get(k[,d]) -> D[k] if k in D, else d.  d defaults to None.\n\n        Parameters\n        ----------\n        default : optional\n          Value to return when key is not present. `None` by default.\n        get_all : bool, optional\n          If True, return all values of multiple identical configuration keys.\n          By default only the last specified value is returned.\n        \"\"\"\n        try:\n            val = self[key]\n            if get_all or not isinstance(val, tuple):\n                return val\n            else:\n                return val[-1]\n        except KeyError:\n            # return as-is, default could be a tuple, hence do not subject to\n            # get_all processing\n            return default\n\n    def get_from_source(self, source, key, default=None):\n        \"\"\"Like get(), but a source can be specific.\n\n        If `source` is 'dataset', only the commited configuration is queried,\n        overrides are applied. In the case of 'local', the committed\n        configuration is ignored, but overrides and configuration from\n        environment variables are applied as usual.\n        \"\"\"\n        if source not in ('dataset', 'local'):\n            raise ValueError(\"source must be 'dataset' or 'local'\")\n        if source == 'dataset':\n            return self.overrides.get(\n                key,\n                self._stores['dataset']['cfg'].get(\n                    key,\n                    default))\n        else:\n            if key not in self._stores['dataset']['cfg']:\n                # the key is not in the committed config, hence we can\n                # just report based on the merged representation\n                return self.get(key, default)\n            else:\n                # expensive case, rebuild a config without the committed\n                # dataset config contributing\n                env = {}\n                _update_from_env(env)\n                return env.get(\n                    key,\n                    self.overrides.get(\n                        key,\n                        self._stores['local']['cfg'].get(\n                            key,\n                            default)))\n\n    #\n    # Compatibility with ConfigParser API\n    #\n    def sections(self):\n        \"\"\"Returns a list of the sections available\"\"\"\n        return list(set([cfg_section_regex.match(k).group(1) for k in self._merged_store]))\n\n    def options(self, section):\n        \"\"\"Returns a list of options available in the specified section.\"\"\"\n        opts = []\n        for k in self._merged_store:\n            sec, opt = cfg_sectionoption_regex.match(k).groups()\n            if sec == section:\n                opts.append(opt)\n        return opts\n\n    def has_section(self, section):\n        \"\"\"Indicates whether a section is present in the configuration\"\"\"\n        for k in self._merged_store:\n            if k.startswith(section):\n                return True\n        return False\n\n    def has_option(self, section, option):\n        \"\"\"If the given section exists, and contains the given option\"\"\"\n        for k in self._merged_store:\n            sec, opt = cfg_sectionoption_regex.match(k).groups()\n            if sec == section and opt == option:\n                return True\n        return False\n\n    def _get_type(self, typefn, section, option):\n        key = '.'.join([section, option])\n        # Mimic the handling of get_value(..., default=None), while still going\n        # through get() in order to get its default tuple handling.\n        if key not in self:\n            raise KeyError(key)\n        return typefn(self.get(key))\n\n    def getint(self, section, option):\n        \"\"\"A convenience method which coerces the option value to an integer\"\"\"\n        return self._get_type(int, section, option)\n\n    def getfloat(self, section, option):\n        \"\"\"A convenience method which coerces the option value to a float\"\"\"\n        return self._get_type(float, section, option)\n\n    def getbool(self, section, option, default=None):\n        \"\"\"A convenience method which coerces the option value to a bool\n\n        Values \"on\", \"yes\", \"true\" and any int!=0 are considered True\n        Values which evaluate to bool False, \"off\", \"no\", \"false\" are considered\n        False\n        TypeError is raised for other values.\n        \"\"\"\n        key = '.'.join([section, option])\n        # Mimic the handling of get_value(..., default=None), while still going\n        # through get() in order to get its default tuple handling.\n        if default is None and key not in self:\n            raise KeyError(key)\n        val = self.get(key, default=default)\n        if val is None:  # no value at all, git treats it as True\n            return True\n        return anything2bool(val)\n\n    # this is a hybrid of ConfigParser and dict API\n    def items(self, section=None):\n        \"\"\"Return a list of (name, value) pairs for each option\n\n        Optionally limited to a given section.\n        \"\"\"\n        if section is None:\n            return self._merged_store.items()\n        return [(k, v) for k, v in self._merged_store.items()\n                if cfg_section_regex.match(k).group(1) == section]\n\n    #\n    # Compatibility with GitPython's ConfigParser\n    #\n    def get_value(self, section, option, default=None):\n        \"\"\"Like `get()`, but with an optional default value\n\n        If the default is not None, the given default value will be returned in\n        case the option did not exist. This behavior imitates GitPython's\n        config parser.\n        \"\"\"\n        try:\n            return self['.'.join((section, option))]\n        except KeyError as e:\n            # this strange dance is needed because gitpython does it this way\n            if default is not None:\n                return default\n            else:\n                raise e\n\n    #\n    # Modify configuration (proxy respective git-config call)\n    #\n    @_where_reload\n    def _run(self, args, where=None, reload=False, **kwargs):\n        \"\"\"Centralized helper to run \"git config\" calls\n\n        Parameters\n        ----------\n        args : list\n          Arguments to pass for git config\n        %s\n        **kwargs\n          Keywords arguments for Runner's call\n        \"\"\"\n        if where:\n            args = self._get_location_args(where) + args\n        if '-l' in args:\n            # we are just reading, no need to reload, no need to lock\n            out = self._runner.run(self._config_cmd + args, **kwargs)\n            return out['stdout'], out['stderr']\n\n        # all other calls are modifications\n        if '--file' in args:\n            # all paths we are passing are absolute\n            custom_file = Path(args[args.index('--file') + 1])\n            custom_file.parent.mkdir(exist_ok=True)\n        lockfile = None\n        if self._repo_dot_git and ('--local' in args or '--file' in args):\n            # modification of config in a dataset\n            lockfile = self._repo_dot_git / 'config.dataladlock'\n        else:\n            # follow pattern in downloaders for lockfile location\n            lockfile = Path(self.obtain('datalad.locations.cache')) \\\n                / 'locks' / 'gitconfig.lck'\n\n        with ConfigManager._run_lock, InterProcessLock(lockfile, logger=lgr):\n            out = self._runner.run(self._config_cmd + args, **kwargs)\n\n        if reload:\n            self.reload()\n        return out['stdout'], out['stderr']\n\n    def _get_location_args(self, where, args=None):\n        if args is None:\n            args = []\n        cfg_labels = ('dataset', 'local', 'global', 'override')\n        if where not in cfg_labels:\n            raise ValueError(\n                \"unknown configuration label '{}' (not in {})\".format(\n                    where, cfg_labels))\n        if where == 'dataset':\n            if not self._repo_pathobj:\n                raise ValueError(\n                    'ConfigManager cannot store configuration to dataset, '\n                    'none specified')\n            dataset_cfgfile = self._repo_pathobj / DATASET_CONFIG_FILE\n            args.extend(['--file', str(dataset_cfgfile)])\n        elif where == 'global':\n            args.append('--global')\n        elif where == 'local':\n            args.append('--local')\n        return args\n\n    @_where_reload\n    def add(self, var, value, where='dataset', reload=True):\n        \"\"\"Add a configuration variable and value\n\n        Parameters\n        ----------\n        var : str\n          Variable name including any section like `git config` expects them, e.g.\n          'core.editor'\n        value : str\n          Variable value\n        %s\"\"\"\n        if where == 'override':\n            from datalad.utils import ensure_list\n            val = ensure_list(self.overrides.pop(var, None))\n            val.append(value)\n            self.overrides[var] = val[0] if len(val) == 1 else val\n            if reload:\n                self.reload(force=True)\n            return\n\n        self._run(['--add', var, value], where=where, reload=reload,\n                  protocol=StdOutErrCapture)\n\n    @_where_reload\n    def set(self, var, value, where='dataset', reload=True, force=False):\n        \"\"\"Set a variable to a value.\n\n        In opposition to `add`, this replaces the value of `var` if there is\n        one already.\n\n        Parameters\n        ----------\n        var : str\n          Variable name including any section like `git config` expects them, e.g.\n          'core.editor'\n        value : str\n          Variable value\n        force: bool\n          if set, replaces all occurrences of `var` by a single one with the\n          given `value`. Otherwise raise if multiple entries for `var` exist\n          already\n        %s\"\"\"\n        if where == 'override':\n            self.overrides[var] = value\n            if reload:\n                self.reload(force=True)\n            return\n\n        from datalad.support.gitrepo import to_options\n\n        self._run(to_options(replace_all=force) + [var, value],\n                  where=where, reload=reload, protocol=StdOutErrCapture)\n\n    @_where_reload\n    def rename_section(self, old, new, where='dataset', reload=True):\n        \"\"\"Rename a configuration section\n\n        Parameters\n        ----------\n        old : str\n          Name of the section to rename.\n        new : str\n          Name of the section to rename to.\n        %s\"\"\"\n        if where == 'override':\n            self.overrides = {\n                (new + k[len(old):]) if k.startswith(old + '.') else k: v\n                for k, v in self.overrides.items()\n            }\n            if reload:\n                self.reload(force=True)\n            return\n\n        self._run(['--rename-section', old, new], where=where, reload=reload)\n\n    @_where_reload\n    def remove_section(self, sec, where='dataset', reload=True):\n        \"\"\"Rename a configuration section\n\n        Parameters\n        ----------\n        sec : str\n          Name of the section to remove.\n        %s\"\"\"\n        if where == 'override':\n            self.overrides = {\n                k: v\n                for k, v in self.overrides.items()\n                if not k.startswith(sec + '.')\n            }\n            if reload:\n                self.reload(force=True)\n            return\n\n        self._run(['--remove-section', sec], where=where, reload=reload)\n\n    @_where_reload\n    def unset(self, var, where='dataset', reload=True):\n        \"\"\"Remove all occurrences of a variable\n\n        Parameters\n        ----------\n        var : str\n          Name of the variable to remove\n        %s\"\"\"\n        if where == 'override':\n            self.overrides.pop(var, None)\n            if reload:\n                self.reload(force=True)\n            return\n\n        # use unset all as it is simpler for now\n        self._run(['--unset-all', var], where=where, reload=reload)\n\n\ndef rewrite_url(cfg, url):\n    \"\"\"Any matching 'url.<base>.insteadOf' configuration is applied\n\n    Any URL that starts with such a configuration will be rewritten\n    to start, instead, with <base>. When more than one insteadOf\n    strings match a given URL, the longest match is used.\n\n    Parameters\n    ----------\n    cfg : ConfigManager or dict\n      dict-like with configuration variable name/value-pairs.\n    url : str\n      URL to be rewritten, if matching configuration is found.\n\n    Returns\n    -------\n    str\n      Rewritten or unmodified URL.\n    \"\"\"\n    insteadof = {\n        # only leave the base url\n        k[4:-10]: v\n        for k, v in cfg.items()\n        if k.startswith('url.') and k.endswith('.insteadof')\n    }\n\n    # all config that applies\n    matches = {\n        key: v\n        for key, val in insteadof.items()\n        for v in (val if isinstance(val, tuple) else (val,))\n        if url.startswith(v)\n    }\n    # find longest match, like Git does\n    if matches:\n        rewrite_base, match = sorted(\n            matches.items(),\n            key=lambda x: len(x[1]),\n            reverse=True,\n        )[0]\n        if sum(match == v for v in matches.values()) > 1:\n            lgr.warning(\n                \"Ignoring URL rewrite configuration for '%s', \"\n                \"multiple conflicting definitions exists: %s\",\n                match,\n                ['url.{}.insteadof'.format(k)\n                 for k, v in matches.items()\n                 if v == match]\n            )\n        else:\n            url = '{}{}'.format(rewrite_base, url[len(match):])\n    return url\n\n\n# for convenience, bind to class too\nConfigManager.rewrite_url = rewrite_url\n\n#\n# Helpers for bypassing git-config when _writing_ config items,\n# mostly useful when a large number of changes needs to be made\n# and directly file manipulation without a safety net is worth\n# the risk for performance reasons.\n#\n\ndef quote_config(v):\n    \"\"\"Helper to perform minimal quoting of config keys/value parts\n\n    Parameters\n    ----------\n    v : str\n      To-be-quoted string\n    \"\"\"\n    white = (' ', '\\t')\n    # backslashes need to be quoted in any case\n    v = v.replace('\\\\', '\\\\\\\\')\n    # must not have additional unquoted quotes\n    v = v.replace('\"', '\\\\\"')\n    if v[0] in white or v[-1] in white:\n        # quoting the value due to leading/trailing whitespace\n        v = '\"{}\"'.format(v)\n    return v\n\n\ndef write_config_section(fobj, suite, name, props):\n    \"\"\"Write a config section with (multiple) settings.\n\n    Parameters\n    ----------\n    fobj : File\n       Opened target file\n    suite : str\n       First item of the section name, e.g. 'submodule', or\n       'datalad'\n    name : str\n       Remainder of the section name\n    props : dict\n       Keys are configuration setting names within the section\n       context (i.e. not duplicating `suite` and/or `name`, values\n       are configuration setting values.\n    \"\"\"\n    fmt = '[{_suite_} {_q_}{_name_}{_q_}]\\n'\n    for p in props:\n        fmt += '\\t{p} = {{{p}}}\\n'.format(p=p)\n    quoted_name = quote_config(name)\n    fobj.write(\n        fmt.format(\n            _suite_=suite,\n            _q_='' if quoted_name.startswith('\"') else '\"',\n            _name_=quoted_name,\n            **{k: quote_config(v) for k, v in props.items()}))\n",
            "header": {
                "file": {
                    "path": "datalad/config.py"
                },
                "misc_line": null
            },
            "index_line": {
                "old_commit_hash": "204eae6fc",
                "new_commit_hash": "6ae65ddc2",
                "mode": "100644"
            },
            "is_binary_file": false,
            "binary_line": null,
            "minus_file": {
                "path": "a/datalad/config.py"
            },
            "plus_file": {
                "path": "b/datalad/config.py"
            },
            "hunks": [
                {
                    "descriptor": {
                        "old_range": {
                            "start": 125,
                            "length": 11
                        },
                        "new_range": {
                            "start": 125,
                            "length": 11
                        },
                        "section": "def _update_from_env(store):"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "    overrides = {}",
                                "type": "context"
                            },
                            {
                                "content": "    dct = {}",
                                "type": "context"
                            },
                            {
                                "content": "    for k in os.environ:",
                                "type": "context"
                            },
                            {
                                "content": "        if k == \"DATALAD_CONFIG_OVERRIDES\":",
                                "type": "deleted"
                            },
                            {
                                "content": "        if k == \"DATALAD_CONFIG_OVERRIDES_JSON\":",
                                "type": "added"
                            },
                            {
                                "content": "            try:",
                                "type": "context"
                            },
                            {
                                "content": "                overrides = json.loads(os.environ[k])",
                                "type": "context"
                            },
                            {
                                "content": "            except json.decoder.JSONDecodeError as exc:",
                                "type": "context"
                            },
                            {
                                "content": "                lgr.warning(\"Failed to load DATALAD_CONFIG_OVERRIDES: %s\",",
                                "type": "deleted"
                            },
                            {
                                "content": "                lgr.warning(\"Failed to load DATALAD_CONFIG_OVERRIDES_JSON: %s\",",
                                "type": "added"
                            },
                            {
                                "content": "                            exc)",
                                "type": "context"
                            },
                            {
                                "content": "        elif k.startswith('DATALAD_'):",
                                "type": "context"
                            },
                            {
                                "content": "            dct[k.replace('__', '-').replace('_', '.').lower()] = os.environ[k]",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [
                        {
                            "file_name": "datalad/config.py",
                            "type": "function",
                            "name": "_update_from_env",
                            "content": "def _update_from_env(store):\n    overrides = {}\n    dct = {}\n    for k in os.environ:\n        if k == \"DATALAD_CONFIG_OVERRIDES\":\n            try:\n                overrides = json.loads(os.environ[k])\n            except json.decoder.JSONDecodeError as exc:\n                lgr.warning(\"Failed to load DATALAD_CONFIG_OVERRIDES: %s\",\n                            exc)\n        elif k.startswith('DATALAD_'):\n            dct[k.replace('__', '-').replace('_', '.').lower()] = os.environ[k]\n    if overrides:\n        store.update(overrides)\n    store.update(dct)",
                            "ast_type_str": "FunctionDef",
                            "start_lineno": 124,
                            "end_lineno": 138,
                            "parent": null
                        }
                    ],
                    "added_entities": [],
                    "deleted_entities": []
                }
            ]
        },
        {
            "old_file_content": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-; coding: utf-8 -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"test command datalad run-procedure\n\n\"\"\"\n\n__docformat__ = 'restructuredtext'\n\nimport os.path as op\nimport sys\nfrom unittest.mock import patch\n\nfrom datalad.cmd import (\n    WitlessRunner,\n    KillOutput,\n)\nfrom datalad.utils import (\n    chpwd,\n    quote_cmdlinearg,\n    swallow_outputs,\n)\nfrom datalad.tests.utils import (\n    OBSCURE_FILENAME,\n    assert_false,\n    assert_in_results,\n    assert_not_in_results,\n    assert_raises,\n    assert_repo_status,\n    assert_result_count,\n    assert_true,\n    eq_,\n    known_failure_windows,\n    ok_file_has_content,\n    on_windows,\n    patch_config,\n    skip_if,\n    with_tempfile,\n    with_tree,\n)\nfrom datalad.distribution.dataset import Dataset\nfrom datalad.support.exceptions import (\n    CommandError,\n    InsufficientArgumentsError,\n)\nfrom datalad.api import run_procedure\nfrom datalad import cfg as dl_cfg\n\n\n@with_tempfile(mkdir=True)\ndef test_invalid_call(path):\n    with chpwd(path):\n        # ^ Change directory so that we don't fail with an\n        # InvalidGitRepositoryError if the test is executed from a git\n        # worktree.\n\n        # needs spec or discover\n        assert_raises(InsufficientArgumentsError, run_procedure)\n        res = run_procedure('unknown', on_failure='ignore')\n        assert_true(len(res) == 1)\n        assert_in_results(res, status=\"impossible\")\n\n\n@known_failure_windows  #FIXME\n@with_tree(tree={'README.md': 'dirty'})\ndef test_dirty(path):\n    ds = Dataset(path).create(force=True)\n    # must fail, because README.md is to be modified, but already dirty\n    assert_raises(CommandError, ds.run_procedure, 'cfg_yoda')\n    # make sure that was the issue\n    # save to git explicitly to keep the test simple and avoid unlocking...\n    ds.save('README.md', to_git=True)\n    ds.run_procedure('cfg_yoda')\n    assert_repo_status(ds.path)\n\n\n@skip_if(cond=on_windows and dl_cfg.obtain(\"datalad.repo.version\") < 6)\n@with_tree(tree={\n    'code': {'datalad_test_proc.py': \"\"\"\\\nimport sys\nimport os.path as op\nfrom datalad.api import save, Dataset\n\nwith open(op.join(sys.argv[1], 'fromproc.txt'), 'w') as f:\n    f.write('hello\\\\n')\nsave(dataset=Dataset(sys.argv[1]), path='fromproc.txt')\n\"\"\"},\n    'cfg_yoda.sh': \"\"\"\\\n#!/bin/bash\necho \"I am never to be ran but could be discovered\"\nexit 1\n\"\"\"})\n@with_tempfile\ndef test_procedure_discovery(path, super_path):\n    with chpwd(path):\n        # ^ Change directory so that we don't fail with an\n        # InvalidGitRepositoryError if the test is executed from a git\n        # worktree.\n        ps = run_procedure(discover=True)\n        # there are a few procedures coming with datalad, needs to find them\n        assert_true(len(ps) > 2)\n        # we get essential properties\n        _check_procedure_properties(ps)\n\n    # set up dataset with registered procedure (c&p from test_basics):\n    ds = Dataset(path).create(force=True)\n    # extra check: must not pick up cfg_yoda.sh in top directory\n    ds.run_procedure('cfg_yoda')\n\n    # path to a procedure which is not under any \"standard\" location but\n    # present in the dataset\n    code_dir_procedure_path = op.join(ds.path, 'code', 'datalad_test_proc.py')\n    top_dir_procedure_path = op.join(ds.path, 'cfg_yoda.sh')\n\n    # run discovery on the dataset:\n    ps = ds.run_procedure(discover=True)\n    # it should not be found magically by default\n    assert_not_in_results(ps, path=code_dir_procedure_path)\n    assert_not_in_results(ps, path=top_dir_procedure_path)\n\n    with patch_config({'datalad.locations.extra-procedures': op.join(ds.path, 'code')}):\n        # run discovery on the dataset:\n        ps = ds.run_procedure(discover=True)\n        # still needs to find procedures coming with datalad\n        assert_true(len(ps) > 3)\n        # and procedure under the path we specified\n        assert_result_count(ps, 1, path=code_dir_procedure_path)\n        assert_not_in_results(ps, path=top_dir_procedure_path)\n\n    # multiple extra locations\n    with patch_config({'datalad.locations.extra-procedures': [op.join(ds.path, 'code'), ds.path]}):\n        # run discovery on the dataset:\n        ps = ds.run_procedure(discover=True)\n        # still needs to find procedures coming with datalad\n        assert_true(len(ps) > 4)\n        # and procedure under the path we specified\n        assert_result_count(ps, 1, path=code_dir_procedure_path)\n        assert_result_count(ps, 1, path=top_dir_procedure_path)\n\n    # configure dataset to look for procedures in its code folder\n    ds.config.add(\n        'datalad.locations.dataset-procedures',\n        'code',\n        where='dataset')\n    ds.save(op.join('.datalad', 'config'))\n\n    # run discovery on the dataset:\n    ps = ds.run_procedure(discover=True)\n\n    # still needs to find procedures coming with datalad\n    assert_true(len(ps) > 2)\n    # we get three essential properties\n    _check_procedure_properties(ps)\n    # dataset's procedure needs to be in the results\n    # and only a single one\n    assert_result_count(ps, 1, path=code_dir_procedure_path)\n    # a subdir shouldn't be considered a procedure just because it's \"executable\"\n    assert_not_in_results(ps, path=op.join(ds.path, 'code', 'testdir'))\n\n    # make it a subdataset and try again:\n    # first we need to save the beast to make install work\n    ds.save()\n    super = Dataset(super_path).create()\n    super.install('sub', source=ds.path)\n\n    ps = super.run_procedure(discover=True)\n    # still needs to find procedures coming with datalad\n    assert_true(len(ps) > 2)\n    _check_procedure_properties(ps)\n    # dataset's procedure needs to be in the results\n    assert_in_results(ps, path=op.join(super.path, 'sub', 'code',\n                                       'datalad_test_proc.py'))\n\n    if not on_windows:  # no symlinks\n        import os\n        # create a procedure which is a broken symlink, but recognizable as a\n        # python script:\n        os.symlink(op.join(super.path, 'sub', 'not_existent'),\n                   op.join(super.path, 'sub', 'code', 'broken_link_proc.py'))\n        # broken symlink at procedure location, but we can't tell, whether it is\n        # an actual procedure without any guess on how to execute it:\n        os.symlink(op.join(super.path, 'sub', 'not_existent'),\n                   op.join(super.path, 'sub', 'code', 'unknwon_broken_link'))\n\n        ps = super.run_procedure(discover=True)\n        # still needs to find procedures coming with datalad and the dataset\n        # procedure registered before\n        assert_true(len(ps) > 3)\n        assert_in_results(ps, path=op.join(super.path, 'sub', 'code',\n                                           'broken_link_proc.py'),\n                          state='absent')\n        assert_in_results(\n            ps,\n            path=op.join(super.path, 'sub', 'code',\n                         'unknwon_broken_link'),\n            state='absent')\n\n\ndef _check_procedure_properties(ps):\n    \"\"\"a common check that we get three essential properties\"\"\"\n    eq_(\n        sum(['procedure_type' in p and\n             'procedure_callfmt' in p and\n             'path' in p\n             for p in ps]),\n        len(ps))\n\n\n@skip_if(cond=on_windows and dl_cfg.obtain(\"datalad.repo.version\") < 6)\n@with_tree(tree={\n    'code': {'datalad_test_proc.py': \"\"\"\\\nimport sys\nimport os.path as op\nfrom datalad.api import save, Dataset\n\nwith open(op.join(sys.argv[1], 'fromproc.txt'), 'w') as f:\n    f.write('{}\\\\n'.format(sys.argv[2]))\nsave(dataset=Dataset(sys.argv[1]), path='fromproc.txt')\n\"\"\"}})\ndef test_configs(path):\n\n    # set up dataset with registered procedure (c&p from test_basics):\n    ds = Dataset(path).create(force=True)\n    ds.run_procedure('cfg_yoda')\n    # configure dataset to look for procedures in its code folder\n    ds.config.add(\n        'datalad.locations.dataset-procedures',\n        'code',\n        where='dataset')\n\n    # 1. run procedure based on execution guessing by run_procedure:\n    ds.run_procedure(spec=['datalad_test_proc', 'some_arg'])\n    # look for traces\n    ok_file_has_content(op.join(ds.path, 'fromproc.txt'), 'some_arg\\n')\n\n    # 2. now configure specific call format including usage of substitution config\n    # for run:\n    ds.config.add(\n        'datalad.procedures.datalad_test_proc.call-format',\n        u'%s {script} {ds} {{mysub}} {args}' % quote_cmdlinearg(sys.executable),\n        where='dataset'\n    )\n    ds.config.add(\n        'datalad.run.substitutions.mysub',\n        'dataset-call-config',\n        where='dataset'\n    )\n    # TODO: Should we allow for --inputs/--outputs arguments for run_procedure\n    #       (to be passed into run)?\n    ds.unlock(\"fromproc.txt\")\n    # run again:\n    ds.run_procedure(spec=['datalad_test_proc', 'some_arg'])\n    # look for traces\n    ok_file_has_content(op.join(ds.path, 'fromproc.txt'), 'dataset-call-config\\n')\n\n    # 3. have a conflicting config at user-level, which should override the\n    # config on dataset level:\n    ds.config.add(\n        'datalad.procedures.datalad_test_proc.call-format',\n        u'%s {script} {ds} local {args}' % quote_cmdlinearg(sys.executable),\n        where='local'\n    )\n    ds.unlock(\"fromproc.txt\")\n    # run again:\n    ds.run_procedure(spec=['datalad_test_proc', 'some_arg'])\n    # look for traces\n    ok_file_has_content(op.join(ds.path, 'fromproc.txt'), 'local\\n')\n\n    # 4. get configured help message:\n    r = ds.run_procedure('datalad_test_proc', help_proc=True,\n                         on_failure='ignore')\n    assert_true(len(r) == 1)\n    assert_in_results(r, status=\"impossible\")\n\n    ds.config.add(\n        'datalad.procedures.datalad_test_proc.help',\n        \"This is a help message\",\n        where='dataset'\n    )\n\n    r = ds.run_procedure('datalad_test_proc', help_proc=True)\n    assert_true(len(r) == 1)\n    assert_in_results(r, message=\"This is a help message\", status='ok')\n\n\n@known_failure_windows\n@with_tree(tree={\n    'code': {'datalad_test_proc.py': \"\"\"\\\nimport sys\nimport os.path as op\nfrom datalad.api import save, Dataset\n\nwith open(op.join(sys.argv[1], sys.argv[2]), 'w') as f:\n    f.write('hello\\\\n')\nsave(dataset=Dataset(sys.argv[1]), path=sys.argv[2])\n\"\"\"}})\ndef test_spaces(path):\n    \"\"\"\n    Test whether args with spaces are correctly parsed.\n    \"\"\"\n    ds = Dataset(path).create(force=True)\n    ds.run_procedure('cfg_yoda')\n    # configure dataset to look for procedures in its code folder\n    ds.config.add(\n        'datalad.locations.dataset-procedures',\n        'code',\n        where='dataset')\n    # 1. run procedure based on execution guessing by run_procedure:\n    ds.run_procedure(spec=['datalad_test_proc', 'with spaces', 'unrelated'])\n    # check whether file has name with spaces\n    ok_file_has_content(op.join(ds.path, 'with spaces'), 'hello\\n')\n\n\n@known_failure_windows\n@with_tree(tree={OBSCURE_FILENAME:\n                 {\"code\": {\"just2args.py\": \"\"\"\nimport sys\nprint(sys.argv)\n# script, dataset, and two others\nassert len(sys.argv) == 4\n\"\"\"}}})\ndef test_quoting(path):\n    ds = Dataset(op.join(path, OBSCURE_FILENAME)).create(force=True)\n    # Our custom procedure fails if it receives anything other than two\n    # procedure arguments (so the script itself receives 3). Check a few cases\n    # from the Python API and CLI.\n    ds.config.add(\"datalad.locations.dataset-procedures\", \"code\",\n                  where=\"dataset\")\n    with swallow_outputs():\n        ds.run_procedure(spec=[\"just2args\", \"with ' sing\", 'with \" doub'])\n        with assert_raises(CommandError):\n            ds.run_procedure(spec=[\"just2args\", \"still-one arg\"])\n\n        runner = WitlessRunner(cwd=ds.path)\n        runner.run(\n            \"datalad run-procedure just2args \\\"with ' sing\\\" 'with \\\" doub'\",\n            protocol=KillOutput)\n        with assert_raises(CommandError):\n            runner.run(\n                \"datalad run-procedure just2args 'still-one arg'\",\n                protocol=KillOutput)\n\n\n@with_tree(tree={\n    # \"TEXT\" ones\n    'empty': '',  # we have special rule to treat empty ones as text\n    # check various structured files - libmagic might change its decisions which\n    # can effect git-annex. https://github.com/datalad/datalad/issues/3361\n    'JSON': \"\"\"\\\n{\n    \"name\": \"John Smith\",\n    \"age\": 33\n}\n\"\"\",\n    'YAML': \"\"\"\\\n--- # The Smiths\n- {name: John Smith, age: 33}\n- name: Mary Smith\n  age: 27\n\"\"\",\n    'MARKDOWN': \"\"\"\\\n# Title\n\n## Section1\n\nWhen the earth was flat\n\n## Section2\n\"\"\",\n    # BINARY ones\n    '0blob': '\\x00',\n    'emptyline': '\\n',  # libmagic: \"binary\" \"application/octet-stream\"\n})\ndef test_text2git(path):\n    # Test if files being correctly annexed in a ds configured with text2git.\n    TEXT_FILES = ('JSON', 'YAML', 'MARKDOWN', 'empty')\n    BINARY_FILES = ('0blob', 'emptyline')\n\n    ds = Dataset(path).create(force=True)\n    ds.run_procedure('cfg_text2git')\n    ds.save(path=TEXT_FILES + BINARY_FILES, message=\"added all files\")\n    assert_repo_status(ds.path)\n\n    # check that text files are not annexed\n    for f in TEXT_FILES:\n        assert_false(ds.repo.is_under_annex(f))\n    # and trivial binaries - annexed\n    for f in BINARY_FILES:\n        assert_true(ds.repo.is_under_annex(f))\n\n\n@with_tree(tree={\".datalad\": {\"procedures\": {\"print_args\": \"\"\"\nimport sys\nprint(sys.argv)\n\"\"\"}}})\ndef test_name_with_underscore(path):\n    ds = Dataset(path).create(force=True)\n\n    # Procedure name with underscore can't be reached directly with a DATALAD_\n    # environment variable.\n    with patch.dict(\"os.environ\",\n                    {\"DATALAD_PROCEDURES_PRINT_ARGS_CALL__FORMAT\":\n                     'python {script}'}):\n        with assert_raises(ValueError):\n            ds.run_procedure(spec=[\"print_args\"])\n\n    # But it can be set via DATALAD_CONFIG_OVERRIDES.\n    with patch.dict(\"os.environ\",\n                    {\"DATALAD_CONFIG_OVERRIDES\":\n                     '{\"datalad.procedures.print_args.call-format\": '\n                     '\"python {script}\"}'}):\n        ds.config.reload()\n        ds.run_procedure(spec=[\"print_args\"])\n",
            "new_file_content": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-; coding: utf-8 -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"test command datalad run-procedure\n\n\"\"\"\n\n__docformat__ = 'restructuredtext'\n\nimport os.path as op\nimport sys\nfrom unittest.mock import patch\n\nfrom datalad.cmd import (\n    WitlessRunner,\n    KillOutput,\n)\nfrom datalad.utils import (\n    chpwd,\n    quote_cmdlinearg,\n    swallow_outputs,\n)\nfrom datalad.tests.utils import (\n    OBSCURE_FILENAME,\n    assert_false,\n    assert_in_results,\n    assert_not_in_results,\n    assert_raises,\n    assert_repo_status,\n    assert_result_count,\n    assert_true,\n    eq_,\n    known_failure_windows,\n    ok_file_has_content,\n    on_windows,\n    patch_config,\n    skip_if,\n    with_tempfile,\n    with_tree,\n)\nfrom datalad.distribution.dataset import Dataset\nfrom datalad.support.exceptions import (\n    CommandError,\n    InsufficientArgumentsError,\n)\nfrom datalad.api import run_procedure\nfrom datalad import cfg as dl_cfg\n\n\n@with_tempfile(mkdir=True)\ndef test_invalid_call(path):\n    with chpwd(path):\n        # ^ Change directory so that we don't fail with an\n        # InvalidGitRepositoryError if the test is executed from a git\n        # worktree.\n\n        # needs spec or discover\n        assert_raises(InsufficientArgumentsError, run_procedure)\n        res = run_procedure('unknown', on_failure='ignore')\n        assert_true(len(res) == 1)\n        assert_in_results(res, status=\"impossible\")\n\n\n@known_failure_windows  #FIXME\n@with_tree(tree={'README.md': 'dirty'})\ndef test_dirty(path):\n    ds = Dataset(path).create(force=True)\n    # must fail, because README.md is to be modified, but already dirty\n    assert_raises(CommandError, ds.run_procedure, 'cfg_yoda')\n    # make sure that was the issue\n    # save to git explicitly to keep the test simple and avoid unlocking...\n    ds.save('README.md', to_git=True)\n    ds.run_procedure('cfg_yoda')\n    assert_repo_status(ds.path)\n\n\n@skip_if(cond=on_windows and dl_cfg.obtain(\"datalad.repo.version\") < 6)\n@with_tree(tree={\n    'code': {'datalad_test_proc.py': \"\"\"\\\nimport sys\nimport os.path as op\nfrom datalad.api import save, Dataset\n\nwith open(op.join(sys.argv[1], 'fromproc.txt'), 'w') as f:\n    f.write('hello\\\\n')\nsave(dataset=Dataset(sys.argv[1]), path='fromproc.txt')\n\"\"\"},\n    'cfg_yoda.sh': \"\"\"\\\n#!/bin/bash\necho \"I am never to be ran but could be discovered\"\nexit 1\n\"\"\"})\n@with_tempfile\ndef test_procedure_discovery(path, super_path):\n    with chpwd(path):\n        # ^ Change directory so that we don't fail with an\n        # InvalidGitRepositoryError if the test is executed from a git\n        # worktree.\n        ps = run_procedure(discover=True)\n        # there are a few procedures coming with datalad, needs to find them\n        assert_true(len(ps) > 2)\n        # we get essential properties\n        _check_procedure_properties(ps)\n\n    # set up dataset with registered procedure (c&p from test_basics):\n    ds = Dataset(path).create(force=True)\n    # extra check: must not pick up cfg_yoda.sh in top directory\n    ds.run_procedure('cfg_yoda')\n\n    # path to a procedure which is not under any \"standard\" location but\n    # present in the dataset\n    code_dir_procedure_path = op.join(ds.path, 'code', 'datalad_test_proc.py')\n    top_dir_procedure_path = op.join(ds.path, 'cfg_yoda.sh')\n\n    # run discovery on the dataset:\n    ps = ds.run_procedure(discover=True)\n    # it should not be found magically by default\n    assert_not_in_results(ps, path=code_dir_procedure_path)\n    assert_not_in_results(ps, path=top_dir_procedure_path)\n\n    with patch_config({'datalad.locations.extra-procedures': op.join(ds.path, 'code')}):\n        # run discovery on the dataset:\n        ps = ds.run_procedure(discover=True)\n        # still needs to find procedures coming with datalad\n        assert_true(len(ps) > 3)\n        # and procedure under the path we specified\n        assert_result_count(ps, 1, path=code_dir_procedure_path)\n        assert_not_in_results(ps, path=top_dir_procedure_path)\n\n    # multiple extra locations\n    with patch_config({'datalad.locations.extra-procedures': [op.join(ds.path, 'code'), ds.path]}):\n        # run discovery on the dataset:\n        ps = ds.run_procedure(discover=True)\n        # still needs to find procedures coming with datalad\n        assert_true(len(ps) > 4)\n        # and procedure under the path we specified\n        assert_result_count(ps, 1, path=code_dir_procedure_path)\n        assert_result_count(ps, 1, path=top_dir_procedure_path)\n\n    # configure dataset to look for procedures in its code folder\n    ds.config.add(\n        'datalad.locations.dataset-procedures',\n        'code',\n        where='dataset')\n    ds.save(op.join('.datalad', 'config'))\n\n    # run discovery on the dataset:\n    ps = ds.run_procedure(discover=True)\n\n    # still needs to find procedures coming with datalad\n    assert_true(len(ps) > 2)\n    # we get three essential properties\n    _check_procedure_properties(ps)\n    # dataset's procedure needs to be in the results\n    # and only a single one\n    assert_result_count(ps, 1, path=code_dir_procedure_path)\n    # a subdir shouldn't be considered a procedure just because it's \"executable\"\n    assert_not_in_results(ps, path=op.join(ds.path, 'code', 'testdir'))\n\n    # make it a subdataset and try again:\n    # first we need to save the beast to make install work\n    ds.save()\n    super = Dataset(super_path).create()\n    super.install('sub', source=ds.path)\n\n    ps = super.run_procedure(discover=True)\n    # still needs to find procedures coming with datalad\n    assert_true(len(ps) > 2)\n    _check_procedure_properties(ps)\n    # dataset's procedure needs to be in the results\n    assert_in_results(ps, path=op.join(super.path, 'sub', 'code',\n                                       'datalad_test_proc.py'))\n\n    if not on_windows:  # no symlinks\n        import os\n        # create a procedure which is a broken symlink, but recognizable as a\n        # python script:\n        os.symlink(op.join(super.path, 'sub', 'not_existent'),\n                   op.join(super.path, 'sub', 'code', 'broken_link_proc.py'))\n        # broken symlink at procedure location, but we can't tell, whether it is\n        # an actual procedure without any guess on how to execute it:\n        os.symlink(op.join(super.path, 'sub', 'not_existent'),\n                   op.join(super.path, 'sub', 'code', 'unknwon_broken_link'))\n\n        ps = super.run_procedure(discover=True)\n        # still needs to find procedures coming with datalad and the dataset\n        # procedure registered before\n        assert_true(len(ps) > 3)\n        assert_in_results(ps, path=op.join(super.path, 'sub', 'code',\n                                           'broken_link_proc.py'),\n                          state='absent')\n        assert_in_results(\n            ps,\n            path=op.join(super.path, 'sub', 'code',\n                         'unknwon_broken_link'),\n            state='absent')\n\n\ndef _check_procedure_properties(ps):\n    \"\"\"a common check that we get three essential properties\"\"\"\n    eq_(\n        sum(['procedure_type' in p and\n             'procedure_callfmt' in p and\n             'path' in p\n             for p in ps]),\n        len(ps))\n\n\n@skip_if(cond=on_windows and dl_cfg.obtain(\"datalad.repo.version\") < 6)\n@with_tree(tree={\n    'code': {'datalad_test_proc.py': \"\"\"\\\nimport sys\nimport os.path as op\nfrom datalad.api import save, Dataset\n\nwith open(op.join(sys.argv[1], 'fromproc.txt'), 'w') as f:\n    f.write('{}\\\\n'.format(sys.argv[2]))\nsave(dataset=Dataset(sys.argv[1]), path='fromproc.txt')\n\"\"\"}})\ndef test_configs(path):\n\n    # set up dataset with registered procedure (c&p from test_basics):\n    ds = Dataset(path).create(force=True)\n    ds.run_procedure('cfg_yoda')\n    # configure dataset to look for procedures in its code folder\n    ds.config.add(\n        'datalad.locations.dataset-procedures',\n        'code',\n        where='dataset')\n\n    # 1. run procedure based on execution guessing by run_procedure:\n    ds.run_procedure(spec=['datalad_test_proc', 'some_arg'])\n    # look for traces\n    ok_file_has_content(op.join(ds.path, 'fromproc.txt'), 'some_arg\\n')\n\n    # 2. now configure specific call format including usage of substitution config\n    # for run:\n    ds.config.add(\n        'datalad.procedures.datalad_test_proc.call-format',\n        u'%s {script} {ds} {{mysub}} {args}' % quote_cmdlinearg(sys.executable),\n        where='dataset'\n    )\n    ds.config.add(\n        'datalad.run.substitutions.mysub',\n        'dataset-call-config',\n        where='dataset'\n    )\n    # TODO: Should we allow for --inputs/--outputs arguments for run_procedure\n    #       (to be passed into run)?\n    ds.unlock(\"fromproc.txt\")\n    # run again:\n    ds.run_procedure(spec=['datalad_test_proc', 'some_arg'])\n    # look for traces\n    ok_file_has_content(op.join(ds.path, 'fromproc.txt'), 'dataset-call-config\\n')\n\n    # 3. have a conflicting config at user-level, which should override the\n    # config on dataset level:\n    ds.config.add(\n        'datalad.procedures.datalad_test_proc.call-format',\n        u'%s {script} {ds} local {args}' % quote_cmdlinearg(sys.executable),\n        where='local'\n    )\n    ds.unlock(\"fromproc.txt\")\n    # run again:\n    ds.run_procedure(spec=['datalad_test_proc', 'some_arg'])\n    # look for traces\n    ok_file_has_content(op.join(ds.path, 'fromproc.txt'), 'local\\n')\n\n    # 4. get configured help message:\n    r = ds.run_procedure('datalad_test_proc', help_proc=True,\n                         on_failure='ignore')\n    assert_true(len(r) == 1)\n    assert_in_results(r, status=\"impossible\")\n\n    ds.config.add(\n        'datalad.procedures.datalad_test_proc.help',\n        \"This is a help message\",\n        where='dataset'\n    )\n\n    r = ds.run_procedure('datalad_test_proc', help_proc=True)\n    assert_true(len(r) == 1)\n    assert_in_results(r, message=\"This is a help message\", status='ok')\n\n\n@known_failure_windows\n@with_tree(tree={\n    'code': {'datalad_test_proc.py': \"\"\"\\\nimport sys\nimport os.path as op\nfrom datalad.api import save, Dataset\n\nwith open(op.join(sys.argv[1], sys.argv[2]), 'w') as f:\n    f.write('hello\\\\n')\nsave(dataset=Dataset(sys.argv[1]), path=sys.argv[2])\n\"\"\"}})\ndef test_spaces(path):\n    \"\"\"\n    Test whether args with spaces are correctly parsed.\n    \"\"\"\n    ds = Dataset(path).create(force=True)\n    ds.run_procedure('cfg_yoda')\n    # configure dataset to look for procedures in its code folder\n    ds.config.add(\n        'datalad.locations.dataset-procedures',\n        'code',\n        where='dataset')\n    # 1. run procedure based on execution guessing by run_procedure:\n    ds.run_procedure(spec=['datalad_test_proc', 'with spaces', 'unrelated'])\n    # check whether file has name with spaces\n    ok_file_has_content(op.join(ds.path, 'with spaces'), 'hello\\n')\n\n\n@known_failure_windows\n@with_tree(tree={OBSCURE_FILENAME:\n                 {\"code\": {\"just2args.py\": \"\"\"\nimport sys\nprint(sys.argv)\n# script, dataset, and two others\nassert len(sys.argv) == 4\n\"\"\"}}})\ndef test_quoting(path):\n    ds = Dataset(op.join(path, OBSCURE_FILENAME)).create(force=True)\n    # Our custom procedure fails if it receives anything other than two\n    # procedure arguments (so the script itself receives 3). Check a few cases\n    # from the Python API and CLI.\n    ds.config.add(\"datalad.locations.dataset-procedures\", \"code\",\n                  where=\"dataset\")\n    with swallow_outputs():\n        ds.run_procedure(spec=[\"just2args\", \"with ' sing\", 'with \" doub'])\n        with assert_raises(CommandError):\n            ds.run_procedure(spec=[\"just2args\", \"still-one arg\"])\n\n        runner = WitlessRunner(cwd=ds.path)\n        runner.run(\n            \"datalad run-procedure just2args \\\"with ' sing\\\" 'with \\\" doub'\",\n            protocol=KillOutput)\n        with assert_raises(CommandError):\n            runner.run(\n                \"datalad run-procedure just2args 'still-one arg'\",\n                protocol=KillOutput)\n\n\n@with_tree(tree={\n    # \"TEXT\" ones\n    'empty': '',  # we have special rule to treat empty ones as text\n    # check various structured files - libmagic might change its decisions which\n    # can effect git-annex. https://github.com/datalad/datalad/issues/3361\n    'JSON': \"\"\"\\\n{\n    \"name\": \"John Smith\",\n    \"age\": 33\n}\n\"\"\",\n    'YAML': \"\"\"\\\n--- # The Smiths\n- {name: John Smith, age: 33}\n- name: Mary Smith\n  age: 27\n\"\"\",\n    'MARKDOWN': \"\"\"\\\n# Title\n\n## Section1\n\nWhen the earth was flat\n\n## Section2\n\"\"\",\n    # BINARY ones\n    '0blob': '\\x00',\n    'emptyline': '\\n',  # libmagic: \"binary\" \"application/octet-stream\"\n})\ndef test_text2git(path):\n    # Test if files being correctly annexed in a ds configured with text2git.\n    TEXT_FILES = ('JSON', 'YAML', 'MARKDOWN', 'empty')\n    BINARY_FILES = ('0blob', 'emptyline')\n\n    ds = Dataset(path).create(force=True)\n    ds.run_procedure('cfg_text2git')\n    ds.save(path=TEXT_FILES + BINARY_FILES, message=\"added all files\")\n    assert_repo_status(ds.path)\n\n    # check that text files are not annexed\n    for f in TEXT_FILES:\n        assert_false(ds.repo.is_under_annex(f))\n    # and trivial binaries - annexed\n    for f in BINARY_FILES:\n        assert_true(ds.repo.is_under_annex(f))\n\n\n@with_tree(tree={\".datalad\": {\"procedures\": {\"print_args\": \"\"\"\nimport sys\nprint(sys.argv)\n\"\"\"}}})\ndef test_name_with_underscore(path):\n    ds = Dataset(path).create(force=True)\n\n    # Procedure name with underscore can't be reached directly with a DATALAD_\n    # environment variable.\n    with patch.dict(\"os.environ\",\n                    {\"DATALAD_PROCEDURES_PRINT_ARGS_CALL__FORMAT\":\n                     'python {script}'}):\n        with assert_raises(ValueError):\n            ds.run_procedure(spec=[\"print_args\"])\n\n    # But it can be set via DATALAD_CONFIG_OVERRIDES_JSON.\n    with patch.dict(\"os.environ\",\n                    {\"DATALAD_CONFIG_OVERRIDES_JSON\":\n                     '{\"datalad.procedures.print_args.call-format\": '\n                     '\"python {script}\"}'}):\n        ds.config.reload()\n        ds.run_procedure(spec=[\"print_args\"])\n",
            "header": {
                "file": {
                    "path": "datalad/interface/tests/test_run_procedure.py"
                },
                "misc_line": null
            },
            "index_line": {
                "old_commit_hash": "9954a9ee6",
                "new_commit_hash": "cf8923dfa",
                "mode": "100644"
            },
            "is_binary_file": false,
            "binary_line": null,
            "minus_file": {
                "path": "a/datalad/interface/tests/test_run_procedure.py"
            },
            "plus_file": {
                "path": "b/datalad/interface/tests/test_run_procedure.py"
            },
            "hunks": [
                {
                    "descriptor": {
                        "old_range": {
                            "start": 409,
                            "length": 9
                        },
                        "new_range": {
                            "start": 409,
                            "length": 9
                        },
                        "section": "def test_name_with_underscore(path):"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "        with assert_raises(ValueError):",
                                "type": "context"
                            },
                            {
                                "content": "            ds.run_procedure(spec=[\"print_args\"])",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "    # But it can be set via DATALAD_CONFIG_OVERRIDES.",
                                "type": "deleted"
                            },
                            {
                                "content": "    # But it can be set via DATALAD_CONFIG_OVERRIDES_JSON.",
                                "type": "added"
                            },
                            {
                                "content": "    with patch.dict(\"os.environ\",",
                                "type": "context"
                            },
                            {
                                "content": "                    {\"DATALAD_CONFIG_OVERRIDES\":",
                                "type": "deleted"
                            },
                            {
                                "content": "                    {\"DATALAD_CONFIG_OVERRIDES_JSON\":",
                                "type": "added"
                            },
                            {
                                "content": "                     '{\"datalad.procedures.print_args.call-format\": '",
                                "type": "context"
                            },
                            {
                                "content": "                     '\"python {script}\"}'}):",
                                "type": "context"
                            },
                            {
                                "content": "        ds.config.reload()",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [
                        {
                            "file_name": "datalad/interface/tests/test_run_procedure.py",
                            "type": "function",
                            "name": "test_name_with_underscore",
                            "content": "def test_name_with_underscore(path):\n    ds = Dataset(path).create(force=True)\n\n    # Procedure name with underscore can't be reached directly with a DATALAD_\n    # environment variable.\n    with patch.dict(\"os.environ\",\n                    {\"DATALAD_PROCEDURES_PRINT_ARGS_CALL__FORMAT\":\n                     'python {script}'}):\n        with assert_raises(ValueError):\n            ds.run_procedure(spec=[\"print_args\"])\n\n    # But it can be set via DATALAD_CONFIG_OVERRIDES.\n    with patch.dict(\"os.environ\",\n                    {\"DATALAD_CONFIG_OVERRIDES\":\n                     '{\"datalad.procedures.print_args.call-format\": '\n                     '\"python {script}\"}'}):\n        ds.config.reload()\n        ds.run_procedure(spec=[\"print_args\"])",
                            "ast_type_str": "FunctionDef",
                            "start_lineno": 401,
                            "end_lineno": 418,
                            "parent": null
                        }
                    ],
                    "added_entities": [],
                    "deleted_entities": []
                }
            ]
        },
        {
            "old_file_content": "# ex: set sts=4 ts=4 sw=4 noet:\n# -*- coding: utf-8 -*-\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"\n\n\"\"\"\n\nimport logging\nimport os\nfrom os.path import exists\nfrom os.path import join as opj\n\nfrom unittest.mock import patch\nfrom datalad.tests.utils import (\n    assert_equal,\n    assert_false,\n    assert_in,\n    assert_not_equal,\n    assert_not_in,\n    assert_raises,\n    assert_true,\n    chpwd,\n    ok_file_has_content,\n    with_tempfile,\n    with_testsui,\n    with_tree,\n)\nfrom datalad.utils import (\n    get_home_envvars,\n    swallow_logs,\n    Path\n)\n\nfrom datalad.distribution.dataset import Dataset\nfrom datalad.api import create\nfrom datalad.config import (\n    ConfigManager,\n    rewrite_url,\n    write_config_section,\n)\nfrom datalad.cmd import CommandError\n\nfrom datalad.support.gitrepo import GitRepo\nfrom datalad import cfg as dl_cfg\n\n\n# XXX tabs are intentional (part of the format)!\n# XXX put back! confuses pep8\n_config_file_content = \"\"\"\\\n[something]\nuser = name=Jane Doe\nuser = email=jd@example.com\nnovalue\nempty =\nmyint = 3\n\n[onemore \"complicated の beast with.dot\"]\nfindme = 5.0\n\"\"\"\n\n_dataset_config_template = {\n    'ds': {\n        '.datalad': {\n            'config': _config_file_content}}}\n\n\n@with_tree(tree=_dataset_config_template)\n@with_tempfile(mkdir=True)\ndef test_something(path, new_home):\n    # will refuse to work on dataset without a dataset\n    assert_raises(ValueError, ConfigManager, source='dataset')\n    # now read the example config\n    cfg = ConfigManager(GitRepo(opj(path, 'ds'), create=True), source='dataset')\n    assert_equal(len(cfg), 5)\n    assert_in('something.user', cfg)\n    # multi-value\n    assert_equal(len(cfg['something.user']), 2)\n    assert_equal(cfg['something.user'], ('name=Jane Doe', 'email=jd@example.com'))\n\n    assert_true(cfg.has_section('something'))\n    assert_false(cfg.has_section('somethingelse'))\n    assert_equal(sorted(cfg.sections()),\n                 [u'onemore.complicated の beast with.dot', 'something'])\n    assert_true(cfg.has_option('something', 'user'))\n    assert_false(cfg.has_option('something', 'us?er'))\n    assert_false(cfg.has_option('some?thing', 'user'))\n    assert_equal(sorted(cfg.options('something')), ['empty', 'myint', 'novalue', 'user'])\n    assert_equal(cfg.options(u'onemore.complicated の beast with.dot'), ['findme'])\n\n    assert_equal(\n        sorted(cfg.items()),\n        [(u'onemore.complicated の beast with.dot.findme', '5.0'),\n         ('something.empty', ''),\n         ('something.myint', '3'),\n         ('something.novalue', None),\n         ('something.user', ('name=Jane Doe', 'email=jd@example.com'))])\n    assert_equal(\n        sorted(cfg.items('something')),\n        [('something.empty', ''),\n         ('something.myint', '3'),\n         ('something.novalue', None),\n         ('something.user', ('name=Jane Doe', 'email=jd@example.com'))])\n\n    # by default get last value only\n    assert_equal(\n        cfg.get('something.user'), 'email=jd@example.com')\n    # but can get all values\n    assert_equal(\n        cfg.get('something.user', get_all=True),\n        ('name=Jane Doe', 'email=jd@example.com'))\n    assert_raises(KeyError, cfg.__getitem__, 'somedthing.user')\n    assert_equal(cfg.getfloat(u'onemore.complicated の beast with.dot', 'findme'), 5.0)\n    assert_equal(cfg.getint('something', 'myint'), 3)\n    assert_equal(cfg.getbool('something', 'myint'), True)\n    # git demands a key without value at all to be used as a flag, thus True\n    assert_equal(cfg.getbool('something', 'novalue'), True)\n    assert_equal(cfg.get('something.novalue'), None)\n    # empty value is False\n    assert_equal(cfg.getbool('something', 'empty'), False)\n    assert_equal(cfg.get('something.empty'), '')\n    assert_equal(cfg.getbool('doesnot', 'exist', default=True), True)\n    assert_raises(TypeError, cfg.getbool, 'something', 'user')\n\n    # gitpython-style access\n    assert_equal(cfg.get('something.myint'), cfg.get_value('something', 'myint'))\n    assert_equal(cfg.get_value('doesnot', 'exist', default='oohaaa'), 'oohaaa')\n    # weired, but that is how it is\n    assert_raises(KeyError, cfg.get_value, 'doesnot', 'exist', default=None)\n\n    # modification follows\n    cfg.add('something.new', 'の')\n    assert_equal(cfg.get('something.new'), u'の')\n    # sections are added on demand\n    cfg.add('unheard.of', 'fame')\n    assert_true(cfg.has_section('unheard.of'))\n    comp = cfg.items('something')\n    cfg.rename_section('something', 'this')\n    assert_true(cfg.has_section('this'))\n    assert_false(cfg.has_section('something'))\n    # direct comparision would fail, because of section prefix\n    assert_equal(len(cfg.items('this')), len(comp))\n    # fail if no such section\n    with swallow_logs():\n        assert_raises(CommandError, cfg.rename_section, 'nothere', 'irrelevant')\n    assert_true(cfg.has_option('this', 'myint'))\n    cfg.unset('this.myint')\n    assert_false(cfg.has_option('this', 'myint'))\n\n    # batch a changes\n    cfg.add('mike.wants.to', 'know', reload=False)\n    assert_false('mike.wants.to' in cfg)\n    cfg.add('mike.wants.to', 'eat')\n    assert_true('mike.wants.to' in cfg)\n    assert_equal(len(cfg['mike.wants.to']), 2)\n\n    # set a new one:\n    cfg.set('mike.should.have', 'known')\n    assert_in('mike.should.have', cfg)\n    assert_equal(cfg['mike.should.have'], 'known')\n    # set an existing one:\n    cfg.set('mike.should.have', 'known better')\n    assert_equal(cfg['mike.should.have'], 'known better')\n    # set, while there are several matching ones already:\n    cfg.add('mike.should.have', 'a meal')\n    assert_equal(len(cfg['mike.should.have']), 2)\n    # raises with force=False\n    assert_raises(CommandError,\n                  cfg.set, 'mike.should.have', 'a beer', force=False)\n    assert_equal(len(cfg['mike.should.have']), 2)\n    # replaces all matching ones with force=True\n    cfg.set('mike.should.have', 'a beer', force=True)\n    assert_equal(cfg['mike.should.have'], 'a beer')\n\n    # fails unknown location\n    assert_raises(ValueError, cfg.add, 'somesuch', 'shit', where='umpalumpa')\n\n    # very carefully test non-local config\n    # so carefully that even in case of bad weather Yarik doesn't find some\n    # lame datalad unittest sections in his precious ~/.gitconfig\n    env = get_home_envvars(new_home)\n    with patch.dict('os.environ',\n                    dict(get_home_envvars(new_home), DATALAD_SNEAKY_ADDITION='ignore')):\n        global_gitconfig = opj(new_home, '.gitconfig')\n        assert(not exists(global_gitconfig))\n        globalcfg = ConfigManager()\n        assert_not_in('datalad.unittest.youcan', globalcfg)\n        assert_in('datalad.sneaky.addition', globalcfg)\n        cfg.add('datalad.unittest.youcan', 'removeme', where='global')\n        assert(exists(global_gitconfig))\n        # it did not go into the dataset's config!\n        assert_not_in('datalad.unittest.youcan', cfg)\n        # does not monitor additions!\n        globalcfg.reload(force=True)\n        assert_in('datalad.unittest.youcan', globalcfg)\n        with swallow_logs():\n            assert_raises(\n                CommandError,\n                globalcfg.unset,\n                'datalad.unittest.youcan',\n                where='local')\n        assert(globalcfg.has_section('datalad.unittest'))\n        globalcfg.unset('datalad.unittest.youcan', where='global')\n        # but after we unset the only value -- that section is no longer listed\n        assert (not globalcfg.has_section('datalad.unittest'))\n        assert_not_in('datalad.unittest.youcan', globalcfg)\n        ok_file_has_content(global_gitconfig, \"\")\n\n    cfg = ConfigManager(\n        Dataset(opj(path, 'ds')),\n        source='dataset',\n        overrides={'datalad.godgiven': True})\n    assert_equal(cfg.get('datalad.godgiven'), True)\n    # setter has no effect\n    cfg.set('datalad.godgiven', 'false')\n    assert_equal(cfg.get('datalad.godgiven'), True)\n\n\n@with_tree(tree={\n    'ds': {\n        '.datalad': {\n            'config': \"\"\"\\\n[crazy]\n    fa = !git remote | xargs -r -I REMOTE /bin/bash -c 'echo I: Fetching from REMOTE && git fetch --prune REMOTE && git fetch -t REMOTE' && [ -d .git/svn ] && bash -c 'echo I: Fetching from SVN && git svn fetch' || : && [ -e .gitmodules ] && bash -c 'echo I: Fetching submodules && git submodule foreach git fa' && [ -d .git/sd ] && bash -c 'echo I: Fetching bugs into sd && git-sd pull --all' || :\n    pa = !git paremotes | tr ' ' '\\\\n'  | xargs -r -l1 git push\n    pt = !git testremotes | tr ' ' '\\\\n'  | xargs -r -l1 -I R git push -f R master\n    ptdry = !git testremotes | tr ' ' '\\\\n'  | xargs -r -l1 -I R git push -f --dry-run R master\n    padry = !git paremotes | tr ' ' '\\\\n' | xargs -r -l1 git push --dry-run\n\"\"\"}}})\ndef test_crazy_cfg(path):\n    cfg = ConfigManager(GitRepo(opj(path, 'ds'), create=True), source='dataset')\n    assert_in('crazy.padry', cfg)\n    # make sure crazy config is not read when in local mode\n    cfg = ConfigManager(Dataset(opj(path, 'ds')), source='local')\n    assert_not_in('crazy.padry', cfg)\n    # it will make it in in 'any' mode though\n    cfg = ConfigManager(Dataset(opj(path, 'ds')), source='any')\n    assert_in('crazy.padry', cfg)\n    # typos in the source mode arg will not have silent side-effects\n    assert_raises(\n        ValueError, ConfigManager, Dataset(opj(path, 'ds')), source='locale')\n\n\n@with_tempfile\ndef test_obtain(path):\n    ds = create(path)\n    cfg = ConfigManager(ds)\n    dummy = 'datalad.test.dummy'\n    # we know nothing and we don't know how to ask\n    assert_raises(RuntimeError, cfg.obtain, dummy)\n    # can report known ones\n    cfg.add(dummy, '5.3')\n    assert_equal(cfg.obtain(dummy), '5.3')\n    # better type\n    assert_equal(cfg.obtain(dummy, valtype=float), 5.3)\n    # don't hide type issues, float doesn't become an int magically\n    assert_raises(ValueError, cfg.obtain, dummy, valtype=int)\n    # inject some prior knowledge\n    from datalad.interface.common_cfg import definitions as cfg_defs\n    cfg_defs[dummy] = dict(type=float)\n    # no we don't need to specify a type anymore\n    assert_equal(cfg.obtain(dummy), 5.3)\n    # but if we remove the value from the config, all magic is gone\n    cfg.unset(dummy)\n    # we know nothing and we don't know how to ask\n    assert_raises(RuntimeError, cfg.obtain, dummy)\n\n    #\n    # test actual interaction\n    #\n    @with_testsui()\n    def ask():\n        # fail on unkown dialog type\n        assert_raises(ValueError, cfg.obtain, dummy, dialog_type='Rorschach_test')\n    ask()\n\n    # ask nicely, and get a value of proper type using the preconfiguration\n    @with_testsui(responses='5.3')\n    def ask():\n        assert_equal(\n            cfg.obtain(dummy, dialog_type='question', text='Tell me'), 5.3)\n    ask()\n\n    # preconfigure even more, to get the most compact call\n    cfg_defs[dummy]['ui'] = ('question', dict(text='tell me', title='Gretchen Frage'))\n\n    @with_testsui(responses='5.3')\n    def ask():\n        assert_equal(cfg.obtain(dummy), 5.3)\n    ask()\n\n    @with_testsui(responses='murks')\n    def ask():\n        assert_raises(ValueError, cfg.obtain, dummy)\n    ask()\n\n    # fail to store when destination is not specified, will not even ask\n    @with_testsui()\n    def ask():\n        assert_raises(ValueError, cfg.obtain, dummy, store=True)\n    ask()\n\n    # but we can preconfigure it\n    cfg_defs[dummy]['destination'] = 'broken'\n\n    @with_testsui(responses='5.3')\n    def ask():\n        assert_raises(ValueError, cfg.obtain, dummy, store=True)\n    ask()\n\n    # fixup destination\n    cfg_defs[dummy]['destination'] = 'dataset'\n\n    @with_testsui(responses='5.3')\n    def ask():\n        assert_equal(cfg.obtain(dummy, store=True), 5.3)\n    ask()\n\n    # now it won't have to ask again\n    @with_testsui()\n    def ask():\n        assert_equal(cfg.obtain(dummy), 5.3)\n    ask()\n\n    # wipe it out again\n    cfg.unset(dummy)\n    assert_not_in(dummy, cfg)\n\n    # XXX cannot figure out how I can simulate a simple <Enter>\n    ## respond with accepting the default\n    #@with_testsui(responses=...)\n    #def ask():\n    #    assert_equal(cfg.obtain(dummy, default=5.3), 5.3)\n    #ask()\n\n\ndef test_from_env():\n    cfg = ConfigManager()\n    assert_not_in('datalad.crazy.cfg', cfg)\n    with patch.dict('os.environ',\n                    {'DATALAD_CRAZY_CFG': 'impossibletoguess'}):\n        cfg.reload()\n        assert_in('datalad.crazy.cfg', cfg)\n        assert_equal(cfg['datalad.crazy.cfg'], 'impossibletoguess')\n        # not in dataset-only mode\n        cfg = ConfigManager(Dataset('nowhere'), source='dataset')\n        assert_not_in('datalad.crazy.cfg', cfg)\n    # check env trumps override\n    cfg = ConfigManager()\n    assert_not_in('datalad.crazy.override', cfg)\n    cfg.set('datalad.crazy.override', 'fromoverride', where='override')\n    cfg.reload()\n    assert_equal(cfg['datalad.crazy.override'], 'fromoverride')\n    with patch.dict('os.environ',\n                    {'DATALAD_CRAZY_OVERRIDE': 'fromenv'}):\n        cfg.reload()\n        assert_equal(cfg['datalad.crazy.override'], 'fromenv')\n\n\ndef test_from_env_overrides():\n    cfg = ConfigManager()\n    assert_not_in(\"datalad.FoO\", cfg)\n\n    # Some details, like case and underscores, cannot be handled by the direct\n    # environment variable mapping.\n    with patch.dict(\"os.environ\",\n                    {\"DATALAD_FOO\": \"val\"}):\n        cfg.reload()\n        assert_not_in(\"datalad.FoO\", cfg)\n        assert_equal(cfg[\"datalad.foo\"], \"val\")\n\n    # But they can be handled via DATALAD_CONFIG_OVERRIDES.\n    with patch.dict(\"os.environ\",\n                    {\"DATALAD_CONFIG_OVERRIDES\": '{\"datalad.FoO\": \"val\"}'}):\n        cfg.reload()\n        assert_equal(cfg[\"datalad.FoO\"], \"val\")\n\n    # DATALAD_CONFIG_OVERRIDES isn't limited to datalad variables.\n    with patch.dict(\"os.environ\",\n                    {\"DATALAD_CONFIG_OVERRIDES\": '{\"a.b.c\": \"val\"}'}):\n        cfg.reload()\n        assert_equal(cfg[\"a.b.c\"], \"val\")\n\n    # Explicitly provided DATALAD_ variables take precedence over those in\n    # DATALAD_CONFIG_OVERRIDES.\n    with patch.dict(\"os.environ\",\n                    {\"DATALAD_CONFIG_OVERRIDES\": '{\"datalad.foo\": \"val\"}',\n                     \"DATALAD_FOO\": \"val-direct\"}):\n        cfg.reload()\n        assert_equal(cfg[\"datalad.foo\"], \"val-direct\")\n\n    # JSON decode errors don't lead to crash.\n    with patch.dict(\"os.environ\",\n                    {\"DATALAD_CONFIG_OVERRIDES\": '{'}):\n        with swallow_logs(logging.WARNING) as cml:\n            cfg.reload()\n        assert_in(\"Failed to load DATALAD_CONFIG_OVERRIDE\", cml.out)\n\n\ndef test_overrides():\n    cfg = ConfigManager()\n    # any sensible (and also our CI) test environment(s) should have this\n    assert_in('user.name', cfg)\n    # set\n    cfg.set('user.name', 'myoverride', where='override')\n    assert_equal(cfg['user.name'], 'myoverride')\n    # unset just removes override, not entire config\n    cfg.unset('user.name', where='override')\n    assert_in('user.name', cfg)\n    assert_not_equal('user.name', 'myoverride')\n    # add\n    # there is no initial increment\n    cfg.add('user.name', 'myoverride', where='override')\n    assert_equal(cfg['user.name'], 'myoverride')\n    # same as with add, not a list\n    assert_equal(cfg['user.name'], 'myoverride')\n    # but then there is\n    cfg.add('user.name', 'myother', where='override')\n    assert_equal(cfg['user.name'], ['myoverride', 'myother'])\n    # rename\n    assert_not_in('ups.name', cfg)\n    cfg.rename_section('user', 'ups', where='override')\n    # original variable still there\n    assert_in('user.name', cfg)\n    # rename of override in effect\n    assert_equal(cfg['ups.name'], ['myoverride', 'myother'])\n    # remove entirely by section\n    cfg.remove_section('ups', where='override')\n    from datalad.utils import Path\n    assert_not_in(\n        'ups.name', cfg,\n        (cfg._stores,\n         cfg.overrides,\n    ))\n\n\ndef test_rewrite_url():\n    test_cases = (\n        # no match\n        ('unicorn', 'unicorn'),\n        # custom label replacement\n        ('example:datalad/datalad.git', 'git@example.com:datalad/datalad.git'),\n        # protocol enforcement\n        ('git://example.com/some', 'https://example.com/some'),\n        # multi-match\n        ('mylabel', 'ria+ssh://fully.qualified.com'),\n        ('myotherlabel', 'ria+ssh://fully.qualified.com'),\n        # conflicts, same label pointing to different URLs\n        ('conflict', 'conflict'),\n        # also conflicts, but hidden in a multi-value definition\n        ('conflict2', 'conflict2'),\n    )\n    cfg_in = {\n        # label rewrite\n        'git@example.com:': 'example:',\n        # protocol change\n        'https://example': 'git://example',\n        # multi-value\n        'ria+ssh://fully.qualified.com': ('mylabel', 'myotherlabel'),\n        # conflicting definitions\n        'http://host1': 'conflict',\n        'http://host2': 'conflict',\n        # hidden conflict\n        'http://host3': 'conflict2',\n        'http://host4': ('someokish', 'conflict2'),\n    }\n    cfg = {\n        'url.{}.insteadof'.format(k): v\n        for k, v in cfg_in.items()\n    }\n    for input, output in test_cases:\n        with swallow_logs(logging.WARNING) as msg:\n            assert_equal(rewrite_url(cfg, input), output)\n        if input.startswith('conflict'):\n            assert_in(\"Ignoring URL rewrite\", msg.out)\n\n\n# https://github.com/datalad/datalad/issues/4071\n@with_tempfile()\n@with_tempfile()\ndef test_no_leaks(path1, path2):\n    ds1 = Dataset(path1).create()\n    ds1.config.set('i.was.here', 'today', where='local')\n    assert_in('i.was.here', ds1.config.keys())\n    ds1.config.reload()\n    assert_in('i.was.here', ds1.config.keys())\n    # now we move into this one repo, and create another\n    # make sure that no config from ds1 leaks into ds2\n    with chpwd(path1):\n        ds2 = Dataset(path2)\n        assert_not_in('i.was.here', ds2.config.keys())\n        ds2.config.reload()\n        assert_not_in('i.was.here', ds2.config.keys())\n\n        ds2.create()\n        assert_not_in('i.was.here', ds2.config.keys())\n\n        # and that we do not track the wrong files\n        assert_not_in(ds1.pathobj / '.git' / 'config',\n                      ds2.config._stores['git']['files'])\n        assert_not_in(ds1.pathobj / '.datalad' / 'config',\n                      ds2.config._stores['dataset']['files'])\n        # these are the right ones\n        assert_in(ds2.pathobj / '.git' / 'config',\n                  ds2.config._stores['git']['files'])\n        assert_in(ds2.pathobj / '.datalad' / 'config',\n                  ds2.config._stores['dataset']['files'])\n\n\n@with_tempfile()\ndef test_no_local_write_if_no_dataset(path):\n    Dataset(path).create()\n    with chpwd(path):\n        cfg = ConfigManager()\n        with assert_raises(CommandError):\n            cfg.set('a.b.c', 'd', where='local')\n\n\n@with_tempfile\ndef test_dataset_local_mode(path):\n    ds = create(path)\n    # any sensible (and also our CI) test environment(s) should have this\n    assert_in('user.name', ds.config)\n    # from .datalad/config\n    assert_in('datalad.dataset.id', ds.config)\n    # from .git/config\n    assert_in('annex.version', ds.config)\n    # now check that dataset-local mode doesn't have the global piece\n    cfg = ConfigManager(ds, source='dataset-local')\n    assert_not_in('user.name', cfg)\n    assert_in('datalad.dataset.id', cfg)\n    assert_in('annex.version', cfg)\n\n\n# https://github.com/datalad/datalad/issues/4071\n@with_tempfile\ndef test_dataset_systemglobal_mode(path):\n    ds = create(path)\n    # any sensible (and also our CI) test environment(s) should have this\n    assert_in('user.name', ds.config)\n    # from .datalad/config\n    assert_in('datalad.dataset.id', ds.config)\n    # from .git/config\n    assert_in('annex.version', ds.config)\n    with chpwd(path):\n        # now check that no config from a random dataset at PWD is picked up\n        # if not dataset instance was provided\n        cfg = ConfigManager(dataset=None, source='any')\n        assert_in('user.name', cfg)\n        assert_not_in('datalad.dataset.id', cfg)\n        assert_not_in('annex.version', cfg)\n\n\ndef test_global_config():\n\n    # from within tests, global config should be read from faked $HOME (see\n    # setup_package)\n    glb_cfg_file = Path(os.path.expanduser('~')) / '.gitconfig'\n    assert any(glb_cfg_file.samefile(Path(p)) for p in dl_cfg._stores['git']['files'])\n    assert_equal(dl_cfg.get(\"user.name\"), \"DataLad Tester\")\n    assert_equal(dl_cfg.get(\"user.email\"), \"test@example.com\")\n\n\n@with_tempfile()\ndef test_bare(path):\n    # can we handle a bare repo?\n    gr = GitRepo(path, create=True, bare=True)\n    # do we read the correct local config?\n    assert_in(gr.pathobj / 'config', gr.config._stores['git']['files'])\n    # any sensible (and also our CI) test environment(s) should have this\n    assert_in('user.name', gr.config)\n    # not set something that wasn't there\n    obscure_key = 'sec.reallyobscurename!@@.key'\n    assert_not_in(obscure_key, gr.config)\n    # to the local config, which is easily accessible\n    gr.config.set(obscure_key, 'myvalue', where='local')\n    assert_equal(gr.config.get(obscure_key), 'myvalue')\n    # now make sure the config is where we think it is\n    assert_in(obscure_key.split('.')[1], (gr.pathobj / 'config').read_text())\n\n\n@with_tempfile()\ndef test_write_config_section(path):\n    # can we handle a bare repo?\n    gr = GitRepo(path, create=True, bare=True)\n\n    # test cases\n    # first 3 args are write_config_section() parameters\n    # 4th arg is a list with key/value pairs that should end up in a\n    # ConfigManager after a reload\n    testcfg = [\n        ('submodule', 'sub', dict(active='true', url='http://example.com'), [\n            ('submodule.sub.active', 'true'),\n            ('submodule.sub.url', 'http://example.com'),\n        ]),\n        ('submodule', 'sub\"quote', {\"a-b\": '\"quoted\"', 'c': 'with\"quote'}, [\n            ('submodule.sub\"quote.a-b', '\"quoted\"'),\n            ('submodule.sub\"quote.c', 'with\"quote'),\n        ]),\n        ('short', ' s p a c e ', {\"a123\": ' space all over '}, [\n            ('short. s p a c e .a123', ' space all over '),\n        ]),\n    ]\n\n    for tc in testcfg:\n        # using append mode to provoke potential interference by\n        # successive calls\n        with (gr.pathobj / 'config').open('a') as fobj:\n            write_config_section(fobj, tc[0], tc[1], tc[2])\n        gr.config.reload()\n        for testcase in tc[3]:\n            assert_in(testcase[0], gr.config)\n            assert_equal(testcase[1], gr.config[testcase[0]])\n\n\n@with_tempfile()\ndef test_external_modification(path):\n    from datalad.cmd import WitlessRunner as Runner\n    runner = Runner(cwd=path)\n    repo = GitRepo(path, create=True)\n    config = repo.config\n\n    key = 'sec.sub.key'\n    assert_not_in(key, config)\n    config.set(key, '1', where='local')\n    assert_equal(config[key], '1')\n\n    # we pick up the case where we modified so size changed\n    runner.run(['git', 'config', '--local', '--replace-all', key, '10'])\n    # unfortunately we do not react for .get unless reload. But here\n    # we will test if reload is correctly decides to reload without force\n    config.reload()\n    assert_equal(config[key], '10')\n\n    # and no size change\n    runner.run(['git', 'config', '--local', '--replace-all', key, '11'])\n    config.reload()\n    assert_equal(config[key], '11')\n\n",
            "new_file_content": "# ex: set sts=4 ts=4 sw=4 noet:\n# -*- coding: utf-8 -*-\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"\n\n\"\"\"\n\nimport logging\nimport os\nfrom os.path import exists\nfrom os.path import join as opj\n\nfrom unittest.mock import patch\nfrom datalad.tests.utils import (\n    assert_equal,\n    assert_false,\n    assert_in,\n    assert_not_equal,\n    assert_not_in,\n    assert_raises,\n    assert_true,\n    chpwd,\n    ok_file_has_content,\n    with_tempfile,\n    with_testsui,\n    with_tree,\n)\nfrom datalad.utils import (\n    get_home_envvars,\n    swallow_logs,\n    Path\n)\n\nfrom datalad.distribution.dataset import Dataset\nfrom datalad.api import create\nfrom datalad.config import (\n    ConfigManager,\n    rewrite_url,\n    write_config_section,\n)\nfrom datalad.cmd import CommandError\n\nfrom datalad.support.gitrepo import GitRepo\nfrom datalad import cfg as dl_cfg\n\n\n# XXX tabs are intentional (part of the format)!\n# XXX put back! confuses pep8\n_config_file_content = \"\"\"\\\n[something]\nuser = name=Jane Doe\nuser = email=jd@example.com\nnovalue\nempty =\nmyint = 3\n\n[onemore \"complicated の beast with.dot\"]\nfindme = 5.0\n\"\"\"\n\n_dataset_config_template = {\n    'ds': {\n        '.datalad': {\n            'config': _config_file_content}}}\n\n\n@with_tree(tree=_dataset_config_template)\n@with_tempfile(mkdir=True)\ndef test_something(path, new_home):\n    # will refuse to work on dataset without a dataset\n    assert_raises(ValueError, ConfigManager, source='dataset')\n    # now read the example config\n    cfg = ConfigManager(GitRepo(opj(path, 'ds'), create=True), source='dataset')\n    assert_equal(len(cfg), 5)\n    assert_in('something.user', cfg)\n    # multi-value\n    assert_equal(len(cfg['something.user']), 2)\n    assert_equal(cfg['something.user'], ('name=Jane Doe', 'email=jd@example.com'))\n\n    assert_true(cfg.has_section('something'))\n    assert_false(cfg.has_section('somethingelse'))\n    assert_equal(sorted(cfg.sections()),\n                 [u'onemore.complicated の beast with.dot', 'something'])\n    assert_true(cfg.has_option('something', 'user'))\n    assert_false(cfg.has_option('something', 'us?er'))\n    assert_false(cfg.has_option('some?thing', 'user'))\n    assert_equal(sorted(cfg.options('something')), ['empty', 'myint', 'novalue', 'user'])\n    assert_equal(cfg.options(u'onemore.complicated の beast with.dot'), ['findme'])\n\n    assert_equal(\n        sorted(cfg.items()),\n        [(u'onemore.complicated の beast with.dot.findme', '5.0'),\n         ('something.empty', ''),\n         ('something.myint', '3'),\n         ('something.novalue', None),\n         ('something.user', ('name=Jane Doe', 'email=jd@example.com'))])\n    assert_equal(\n        sorted(cfg.items('something')),\n        [('something.empty', ''),\n         ('something.myint', '3'),\n         ('something.novalue', None),\n         ('something.user', ('name=Jane Doe', 'email=jd@example.com'))])\n\n    # by default get last value only\n    assert_equal(\n        cfg.get('something.user'), 'email=jd@example.com')\n    # but can get all values\n    assert_equal(\n        cfg.get('something.user', get_all=True),\n        ('name=Jane Doe', 'email=jd@example.com'))\n    assert_raises(KeyError, cfg.__getitem__, 'somedthing.user')\n    assert_equal(cfg.getfloat(u'onemore.complicated の beast with.dot', 'findme'), 5.0)\n    assert_equal(cfg.getint('something', 'myint'), 3)\n    assert_equal(cfg.getbool('something', 'myint'), True)\n    # git demands a key without value at all to be used as a flag, thus True\n    assert_equal(cfg.getbool('something', 'novalue'), True)\n    assert_equal(cfg.get('something.novalue'), None)\n    # empty value is False\n    assert_equal(cfg.getbool('something', 'empty'), False)\n    assert_equal(cfg.get('something.empty'), '')\n    assert_equal(cfg.getbool('doesnot', 'exist', default=True), True)\n    assert_raises(TypeError, cfg.getbool, 'something', 'user')\n\n    # gitpython-style access\n    assert_equal(cfg.get('something.myint'), cfg.get_value('something', 'myint'))\n    assert_equal(cfg.get_value('doesnot', 'exist', default='oohaaa'), 'oohaaa')\n    # weired, but that is how it is\n    assert_raises(KeyError, cfg.get_value, 'doesnot', 'exist', default=None)\n\n    # modification follows\n    cfg.add('something.new', 'の')\n    assert_equal(cfg.get('something.new'), u'の')\n    # sections are added on demand\n    cfg.add('unheard.of', 'fame')\n    assert_true(cfg.has_section('unheard.of'))\n    comp = cfg.items('something')\n    cfg.rename_section('something', 'this')\n    assert_true(cfg.has_section('this'))\n    assert_false(cfg.has_section('something'))\n    # direct comparision would fail, because of section prefix\n    assert_equal(len(cfg.items('this')), len(comp))\n    # fail if no such section\n    with swallow_logs():\n        assert_raises(CommandError, cfg.rename_section, 'nothere', 'irrelevant')\n    assert_true(cfg.has_option('this', 'myint'))\n    cfg.unset('this.myint')\n    assert_false(cfg.has_option('this', 'myint'))\n\n    # batch a changes\n    cfg.add('mike.wants.to', 'know', reload=False)\n    assert_false('mike.wants.to' in cfg)\n    cfg.add('mike.wants.to', 'eat')\n    assert_true('mike.wants.to' in cfg)\n    assert_equal(len(cfg['mike.wants.to']), 2)\n\n    # set a new one:\n    cfg.set('mike.should.have', 'known')\n    assert_in('mike.should.have', cfg)\n    assert_equal(cfg['mike.should.have'], 'known')\n    # set an existing one:\n    cfg.set('mike.should.have', 'known better')\n    assert_equal(cfg['mike.should.have'], 'known better')\n    # set, while there are several matching ones already:\n    cfg.add('mike.should.have', 'a meal')\n    assert_equal(len(cfg['mike.should.have']), 2)\n    # raises with force=False\n    assert_raises(CommandError,\n                  cfg.set, 'mike.should.have', 'a beer', force=False)\n    assert_equal(len(cfg['mike.should.have']), 2)\n    # replaces all matching ones with force=True\n    cfg.set('mike.should.have', 'a beer', force=True)\n    assert_equal(cfg['mike.should.have'], 'a beer')\n\n    # fails unknown location\n    assert_raises(ValueError, cfg.add, 'somesuch', 'shit', where='umpalumpa')\n\n    # very carefully test non-local config\n    # so carefully that even in case of bad weather Yarik doesn't find some\n    # lame datalad unittest sections in his precious ~/.gitconfig\n    env = get_home_envvars(new_home)\n    with patch.dict('os.environ',\n                    dict(get_home_envvars(new_home), DATALAD_SNEAKY_ADDITION='ignore')):\n        global_gitconfig = opj(new_home, '.gitconfig')\n        assert(not exists(global_gitconfig))\n        globalcfg = ConfigManager()\n        assert_not_in('datalad.unittest.youcan', globalcfg)\n        assert_in('datalad.sneaky.addition', globalcfg)\n        cfg.add('datalad.unittest.youcan', 'removeme', where='global')\n        assert(exists(global_gitconfig))\n        # it did not go into the dataset's config!\n        assert_not_in('datalad.unittest.youcan', cfg)\n        # does not monitor additions!\n        globalcfg.reload(force=True)\n        assert_in('datalad.unittest.youcan', globalcfg)\n        with swallow_logs():\n            assert_raises(\n                CommandError,\n                globalcfg.unset,\n                'datalad.unittest.youcan',\n                where='local')\n        assert(globalcfg.has_section('datalad.unittest'))\n        globalcfg.unset('datalad.unittest.youcan', where='global')\n        # but after we unset the only value -- that section is no longer listed\n        assert (not globalcfg.has_section('datalad.unittest'))\n        assert_not_in('datalad.unittest.youcan', globalcfg)\n        ok_file_has_content(global_gitconfig, \"\")\n\n    cfg = ConfigManager(\n        Dataset(opj(path, 'ds')),\n        source='dataset',\n        overrides={'datalad.godgiven': True})\n    assert_equal(cfg.get('datalad.godgiven'), True)\n    # setter has no effect\n    cfg.set('datalad.godgiven', 'false')\n    assert_equal(cfg.get('datalad.godgiven'), True)\n\n\n@with_tree(tree={\n    'ds': {\n        '.datalad': {\n            'config': \"\"\"\\\n[crazy]\n    fa = !git remote | xargs -r -I REMOTE /bin/bash -c 'echo I: Fetching from REMOTE && git fetch --prune REMOTE && git fetch -t REMOTE' && [ -d .git/svn ] && bash -c 'echo I: Fetching from SVN && git svn fetch' || : && [ -e .gitmodules ] && bash -c 'echo I: Fetching submodules && git submodule foreach git fa' && [ -d .git/sd ] && bash -c 'echo I: Fetching bugs into sd && git-sd pull --all' || :\n    pa = !git paremotes | tr ' ' '\\\\n'  | xargs -r -l1 git push\n    pt = !git testremotes | tr ' ' '\\\\n'  | xargs -r -l1 -I R git push -f R master\n    ptdry = !git testremotes | tr ' ' '\\\\n'  | xargs -r -l1 -I R git push -f --dry-run R master\n    padry = !git paremotes | tr ' ' '\\\\n' | xargs -r -l1 git push --dry-run\n\"\"\"}}})\ndef test_crazy_cfg(path):\n    cfg = ConfigManager(GitRepo(opj(path, 'ds'), create=True), source='dataset')\n    assert_in('crazy.padry', cfg)\n    # make sure crazy config is not read when in local mode\n    cfg = ConfigManager(Dataset(opj(path, 'ds')), source='local')\n    assert_not_in('crazy.padry', cfg)\n    # it will make it in in 'any' mode though\n    cfg = ConfigManager(Dataset(opj(path, 'ds')), source='any')\n    assert_in('crazy.padry', cfg)\n    # typos in the source mode arg will not have silent side-effects\n    assert_raises(\n        ValueError, ConfigManager, Dataset(opj(path, 'ds')), source='locale')\n\n\n@with_tempfile\ndef test_obtain(path):\n    ds = create(path)\n    cfg = ConfigManager(ds)\n    dummy = 'datalad.test.dummy'\n    # we know nothing and we don't know how to ask\n    assert_raises(RuntimeError, cfg.obtain, dummy)\n    # can report known ones\n    cfg.add(dummy, '5.3')\n    assert_equal(cfg.obtain(dummy), '5.3')\n    # better type\n    assert_equal(cfg.obtain(dummy, valtype=float), 5.3)\n    # don't hide type issues, float doesn't become an int magically\n    assert_raises(ValueError, cfg.obtain, dummy, valtype=int)\n    # inject some prior knowledge\n    from datalad.interface.common_cfg import definitions as cfg_defs\n    cfg_defs[dummy] = dict(type=float)\n    # no we don't need to specify a type anymore\n    assert_equal(cfg.obtain(dummy), 5.3)\n    # but if we remove the value from the config, all magic is gone\n    cfg.unset(dummy)\n    # we know nothing and we don't know how to ask\n    assert_raises(RuntimeError, cfg.obtain, dummy)\n\n    #\n    # test actual interaction\n    #\n    @with_testsui()\n    def ask():\n        # fail on unkown dialog type\n        assert_raises(ValueError, cfg.obtain, dummy, dialog_type='Rorschach_test')\n    ask()\n\n    # ask nicely, and get a value of proper type using the preconfiguration\n    @with_testsui(responses='5.3')\n    def ask():\n        assert_equal(\n            cfg.obtain(dummy, dialog_type='question', text='Tell me'), 5.3)\n    ask()\n\n    # preconfigure even more, to get the most compact call\n    cfg_defs[dummy]['ui'] = ('question', dict(text='tell me', title='Gretchen Frage'))\n\n    @with_testsui(responses='5.3')\n    def ask():\n        assert_equal(cfg.obtain(dummy), 5.3)\n    ask()\n\n    @with_testsui(responses='murks')\n    def ask():\n        assert_raises(ValueError, cfg.obtain, dummy)\n    ask()\n\n    # fail to store when destination is not specified, will not even ask\n    @with_testsui()\n    def ask():\n        assert_raises(ValueError, cfg.obtain, dummy, store=True)\n    ask()\n\n    # but we can preconfigure it\n    cfg_defs[dummy]['destination'] = 'broken'\n\n    @with_testsui(responses='5.3')\n    def ask():\n        assert_raises(ValueError, cfg.obtain, dummy, store=True)\n    ask()\n\n    # fixup destination\n    cfg_defs[dummy]['destination'] = 'dataset'\n\n    @with_testsui(responses='5.3')\n    def ask():\n        assert_equal(cfg.obtain(dummy, store=True), 5.3)\n    ask()\n\n    # now it won't have to ask again\n    @with_testsui()\n    def ask():\n        assert_equal(cfg.obtain(dummy), 5.3)\n    ask()\n\n    # wipe it out again\n    cfg.unset(dummy)\n    assert_not_in(dummy, cfg)\n\n    # XXX cannot figure out how I can simulate a simple <Enter>\n    ## respond with accepting the default\n    #@with_testsui(responses=...)\n    #def ask():\n    #    assert_equal(cfg.obtain(dummy, default=5.3), 5.3)\n    #ask()\n\n\ndef test_from_env():\n    cfg = ConfigManager()\n    assert_not_in('datalad.crazy.cfg', cfg)\n    with patch.dict('os.environ',\n                    {'DATALAD_CRAZY_CFG': 'impossibletoguess'}):\n        cfg.reload()\n        assert_in('datalad.crazy.cfg', cfg)\n        assert_equal(cfg['datalad.crazy.cfg'], 'impossibletoguess')\n        # not in dataset-only mode\n        cfg = ConfigManager(Dataset('nowhere'), source='dataset')\n        assert_not_in('datalad.crazy.cfg', cfg)\n    # check env trumps override\n    cfg = ConfigManager()\n    assert_not_in('datalad.crazy.override', cfg)\n    cfg.set('datalad.crazy.override', 'fromoverride', where='override')\n    cfg.reload()\n    assert_equal(cfg['datalad.crazy.override'], 'fromoverride')\n    with patch.dict('os.environ',\n                    {'DATALAD_CRAZY_OVERRIDE': 'fromenv'}):\n        cfg.reload()\n        assert_equal(cfg['datalad.crazy.override'], 'fromenv')\n\n\ndef test_from_env_overrides():\n    cfg = ConfigManager()\n    assert_not_in(\"datalad.FoO\", cfg)\n\n    # Some details, like case and underscores, cannot be handled by the direct\n    # environment variable mapping.\n    with patch.dict(\"os.environ\",\n                    {\"DATALAD_FOO\": \"val\"}):\n        cfg.reload()\n        assert_not_in(\"datalad.FoO\", cfg)\n        assert_equal(cfg[\"datalad.foo\"], \"val\")\n\n    # But they can be handled via DATALAD_CONFIG_OVERRIDES_JSON.\n    with patch.dict(\"os.environ\",\n                    {\"DATALAD_CONFIG_OVERRIDES_JSON\": '{\"datalad.FoO\": \"val\"}'}):\n        cfg.reload()\n        assert_equal(cfg[\"datalad.FoO\"], \"val\")\n\n    # DATALAD_CONFIG_OVERRIDES_JSON isn't limited to datalad variables.\n    with patch.dict(\"os.environ\",\n                    {\"DATALAD_CONFIG_OVERRIDES_JSON\": '{\"a.b.c\": \"val\"}'}):\n        cfg.reload()\n        assert_equal(cfg[\"a.b.c\"], \"val\")\n\n    # Explicitly provided DATALAD_ variables take precedence over those in\n    # DATALAD_CONFIG_OVERRIDES_JSON.\n    with patch.dict(\"os.environ\",\n                    {\"DATALAD_CONFIG_OVERRIDES_JSON\": '{\"datalad.foo\": \"val\"}',\n                     \"DATALAD_FOO\": \"val-direct\"}):\n        cfg.reload()\n        assert_equal(cfg[\"datalad.foo\"], \"val-direct\")\n\n    # JSON decode errors don't lead to crash.\n    with patch.dict(\"os.environ\",\n                    {\"DATALAD_CONFIG_OVERRIDES_JSON\": '{'}):\n        with swallow_logs(logging.WARNING) as cml:\n            cfg.reload()\n        assert_in(\"Failed to load DATALAD_CONFIG_OVERRIDE\", cml.out)\n\n\ndef test_overrides():\n    cfg = ConfigManager()\n    # any sensible (and also our CI) test environment(s) should have this\n    assert_in('user.name', cfg)\n    # set\n    cfg.set('user.name', 'myoverride', where='override')\n    assert_equal(cfg['user.name'], 'myoverride')\n    # unset just removes override, not entire config\n    cfg.unset('user.name', where='override')\n    assert_in('user.name', cfg)\n    assert_not_equal('user.name', 'myoverride')\n    # add\n    # there is no initial increment\n    cfg.add('user.name', 'myoverride', where='override')\n    assert_equal(cfg['user.name'], 'myoverride')\n    # same as with add, not a list\n    assert_equal(cfg['user.name'], 'myoverride')\n    # but then there is\n    cfg.add('user.name', 'myother', where='override')\n    assert_equal(cfg['user.name'], ['myoverride', 'myother'])\n    # rename\n    assert_not_in('ups.name', cfg)\n    cfg.rename_section('user', 'ups', where='override')\n    # original variable still there\n    assert_in('user.name', cfg)\n    # rename of override in effect\n    assert_equal(cfg['ups.name'], ['myoverride', 'myother'])\n    # remove entirely by section\n    cfg.remove_section('ups', where='override')\n    from datalad.utils import Path\n    assert_not_in(\n        'ups.name', cfg,\n        (cfg._stores,\n         cfg.overrides,\n    ))\n\n\ndef test_rewrite_url():\n    test_cases = (\n        # no match\n        ('unicorn', 'unicorn'),\n        # custom label replacement\n        ('example:datalad/datalad.git', 'git@example.com:datalad/datalad.git'),\n        # protocol enforcement\n        ('git://example.com/some', 'https://example.com/some'),\n        # multi-match\n        ('mylabel', 'ria+ssh://fully.qualified.com'),\n        ('myotherlabel', 'ria+ssh://fully.qualified.com'),\n        # conflicts, same label pointing to different URLs\n        ('conflict', 'conflict'),\n        # also conflicts, but hidden in a multi-value definition\n        ('conflict2', 'conflict2'),\n    )\n    cfg_in = {\n        # label rewrite\n        'git@example.com:': 'example:',\n        # protocol change\n        'https://example': 'git://example',\n        # multi-value\n        'ria+ssh://fully.qualified.com': ('mylabel', 'myotherlabel'),\n        # conflicting definitions\n        'http://host1': 'conflict',\n        'http://host2': 'conflict',\n        # hidden conflict\n        'http://host3': 'conflict2',\n        'http://host4': ('someokish', 'conflict2'),\n    }\n    cfg = {\n        'url.{}.insteadof'.format(k): v\n        for k, v in cfg_in.items()\n    }\n    for input, output in test_cases:\n        with swallow_logs(logging.WARNING) as msg:\n            assert_equal(rewrite_url(cfg, input), output)\n        if input.startswith('conflict'):\n            assert_in(\"Ignoring URL rewrite\", msg.out)\n\n\n# https://github.com/datalad/datalad/issues/4071\n@with_tempfile()\n@with_tempfile()\ndef test_no_leaks(path1, path2):\n    ds1 = Dataset(path1).create()\n    ds1.config.set('i.was.here', 'today', where='local')\n    assert_in('i.was.here', ds1.config.keys())\n    ds1.config.reload()\n    assert_in('i.was.here', ds1.config.keys())\n    # now we move into this one repo, and create another\n    # make sure that no config from ds1 leaks into ds2\n    with chpwd(path1):\n        ds2 = Dataset(path2)\n        assert_not_in('i.was.here', ds2.config.keys())\n        ds2.config.reload()\n        assert_not_in('i.was.here', ds2.config.keys())\n\n        ds2.create()\n        assert_not_in('i.was.here', ds2.config.keys())\n\n        # and that we do not track the wrong files\n        assert_not_in(ds1.pathobj / '.git' / 'config',\n                      ds2.config._stores['git']['files'])\n        assert_not_in(ds1.pathobj / '.datalad' / 'config',\n                      ds2.config._stores['dataset']['files'])\n        # these are the right ones\n        assert_in(ds2.pathobj / '.git' / 'config',\n                  ds2.config._stores['git']['files'])\n        assert_in(ds2.pathobj / '.datalad' / 'config',\n                  ds2.config._stores['dataset']['files'])\n\n\n@with_tempfile()\ndef test_no_local_write_if_no_dataset(path):\n    Dataset(path).create()\n    with chpwd(path):\n        cfg = ConfigManager()\n        with assert_raises(CommandError):\n            cfg.set('a.b.c', 'd', where='local')\n\n\n@with_tempfile\ndef test_dataset_local_mode(path):\n    ds = create(path)\n    # any sensible (and also our CI) test environment(s) should have this\n    assert_in('user.name', ds.config)\n    # from .datalad/config\n    assert_in('datalad.dataset.id', ds.config)\n    # from .git/config\n    assert_in('annex.version', ds.config)\n    # now check that dataset-local mode doesn't have the global piece\n    cfg = ConfigManager(ds, source='dataset-local')\n    assert_not_in('user.name', cfg)\n    assert_in('datalad.dataset.id', cfg)\n    assert_in('annex.version', cfg)\n\n\n# https://github.com/datalad/datalad/issues/4071\n@with_tempfile\ndef test_dataset_systemglobal_mode(path):\n    ds = create(path)\n    # any sensible (and also our CI) test environment(s) should have this\n    assert_in('user.name', ds.config)\n    # from .datalad/config\n    assert_in('datalad.dataset.id', ds.config)\n    # from .git/config\n    assert_in('annex.version', ds.config)\n    with chpwd(path):\n        # now check that no config from a random dataset at PWD is picked up\n        # if not dataset instance was provided\n        cfg = ConfigManager(dataset=None, source='any')\n        assert_in('user.name', cfg)\n        assert_not_in('datalad.dataset.id', cfg)\n        assert_not_in('annex.version', cfg)\n\n\ndef test_global_config():\n\n    # from within tests, global config should be read from faked $HOME (see\n    # setup_package)\n    glb_cfg_file = Path(os.path.expanduser('~')) / '.gitconfig'\n    assert any(glb_cfg_file.samefile(Path(p)) for p in dl_cfg._stores['git']['files'])\n    assert_equal(dl_cfg.get(\"user.name\"), \"DataLad Tester\")\n    assert_equal(dl_cfg.get(\"user.email\"), \"test@example.com\")\n\n\n@with_tempfile()\ndef test_bare(path):\n    # can we handle a bare repo?\n    gr = GitRepo(path, create=True, bare=True)\n    # do we read the correct local config?\n    assert_in(gr.pathobj / 'config', gr.config._stores['git']['files'])\n    # any sensible (and also our CI) test environment(s) should have this\n    assert_in('user.name', gr.config)\n    # not set something that wasn't there\n    obscure_key = 'sec.reallyobscurename!@@.key'\n    assert_not_in(obscure_key, gr.config)\n    # to the local config, which is easily accessible\n    gr.config.set(obscure_key, 'myvalue', where='local')\n    assert_equal(gr.config.get(obscure_key), 'myvalue')\n    # now make sure the config is where we think it is\n    assert_in(obscure_key.split('.')[1], (gr.pathobj / 'config').read_text())\n\n\n@with_tempfile()\ndef test_write_config_section(path):\n    # can we handle a bare repo?\n    gr = GitRepo(path, create=True, bare=True)\n\n    # test cases\n    # first 3 args are write_config_section() parameters\n    # 4th arg is a list with key/value pairs that should end up in a\n    # ConfigManager after a reload\n    testcfg = [\n        ('submodule', 'sub', dict(active='true', url='http://example.com'), [\n            ('submodule.sub.active', 'true'),\n            ('submodule.sub.url', 'http://example.com'),\n        ]),\n        ('submodule', 'sub\"quote', {\"a-b\": '\"quoted\"', 'c': 'with\"quote'}, [\n            ('submodule.sub\"quote.a-b', '\"quoted\"'),\n            ('submodule.sub\"quote.c', 'with\"quote'),\n        ]),\n        ('short', ' s p a c e ', {\"a123\": ' space all over '}, [\n            ('short. s p a c e .a123', ' space all over '),\n        ]),\n    ]\n\n    for tc in testcfg:\n        # using append mode to provoke potential interference by\n        # successive calls\n        with (gr.pathobj / 'config').open('a') as fobj:\n            write_config_section(fobj, tc[0], tc[1], tc[2])\n        gr.config.reload()\n        for testcase in tc[3]:\n            assert_in(testcase[0], gr.config)\n            assert_equal(testcase[1], gr.config[testcase[0]])\n\n\n@with_tempfile()\ndef test_external_modification(path):\n    from datalad.cmd import WitlessRunner as Runner\n    runner = Runner(cwd=path)\n    repo = GitRepo(path, create=True)\n    config = repo.config\n\n    key = 'sec.sub.key'\n    assert_not_in(key, config)\n    config.set(key, '1', where='local')\n    assert_equal(config[key], '1')\n\n    # we pick up the case where we modified so size changed\n    runner.run(['git', 'config', '--local', '--replace-all', key, '10'])\n    # unfortunately we do not react for .get unless reload. But here\n    # we will test if reload is correctly decides to reload without force\n    config.reload()\n    assert_equal(config[key], '10')\n\n    # and no size change\n    runner.run(['git', 'config', '--local', '--replace-all', key, '11'])\n    config.reload()\n    assert_equal(config[key], '11')\n",
            "header": {
                "file": {
                    "path": "datalad/tests/test_config.py"
                },
                "misc_line": null
            },
            "index_line": {
                "old_commit_hash": "db6f5c9b8",
                "new_commit_hash": "ab8f9eeed",
                "mode": "100644"
            },
            "is_binary_file": false,
            "binary_line": null,
            "minus_file": {
                "path": "a/datalad/tests/test_config.py"
            },
            "plus_file": {
                "path": "b/datalad/tests/test_config.py"
            },
            "hunks": [
                {
                    "descriptor": {
                        "old_range": {
                            "start": 373,
                            "length": 29
                        },
                        "new_range": {
                            "start": 373,
                            "length": 29
                        },
                        "section": "def test_from_env_overrides():"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "        assert_not_in(\"datalad.FoO\", cfg)",
                                "type": "context"
                            },
                            {
                                "content": "        assert_equal(cfg[\"datalad.foo\"], \"val\")",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "    # But they can be handled via DATALAD_CONFIG_OVERRIDES.",
                                "type": "deleted"
                            },
                            {
                                "content": "    # But they can be handled via DATALAD_CONFIG_OVERRIDES_JSON.",
                                "type": "added"
                            },
                            {
                                "content": "    with patch.dict(\"os.environ\",",
                                "type": "context"
                            },
                            {
                                "content": "                    {\"DATALAD_CONFIG_OVERRIDES\": '{\"datalad.FoO\": \"val\"}'}):",
                                "type": "deleted"
                            },
                            {
                                "content": "                    {\"DATALAD_CONFIG_OVERRIDES_JSON\": '{\"datalad.FoO\": \"val\"}'}):",
                                "type": "added"
                            },
                            {
                                "content": "        cfg.reload()",
                                "type": "context"
                            },
                            {
                                "content": "        assert_equal(cfg[\"datalad.FoO\"], \"val\")",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "    # DATALAD_CONFIG_OVERRIDES isn't limited to datalad variables.",
                                "type": "deleted"
                            },
                            {
                                "content": "    # DATALAD_CONFIG_OVERRIDES_JSON isn't limited to datalad variables.",
                                "type": "added"
                            },
                            {
                                "content": "    with patch.dict(\"os.environ\",",
                                "type": "context"
                            },
                            {
                                "content": "                    {\"DATALAD_CONFIG_OVERRIDES\": '{\"a.b.c\": \"val\"}'}):",
                                "type": "deleted"
                            },
                            {
                                "content": "                    {\"DATALAD_CONFIG_OVERRIDES_JSON\": '{\"a.b.c\": \"val\"}'}):",
                                "type": "added"
                            },
                            {
                                "content": "        cfg.reload()",
                                "type": "context"
                            },
                            {
                                "content": "        assert_equal(cfg[\"a.b.c\"], \"val\")",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "    # Explicitly provided DATALAD_ variables take precedence over those in",
                                "type": "context"
                            },
                            {
                                "content": "    # DATALAD_CONFIG_OVERRIDES.",
                                "type": "deleted"
                            },
                            {
                                "content": "    # DATALAD_CONFIG_OVERRIDES_JSON.",
                                "type": "added"
                            },
                            {
                                "content": "    with patch.dict(\"os.environ\",",
                                "type": "context"
                            },
                            {
                                "content": "                    {\"DATALAD_CONFIG_OVERRIDES\": '{\"datalad.foo\": \"val\"}',",
                                "type": "deleted"
                            },
                            {
                                "content": "                    {\"DATALAD_CONFIG_OVERRIDES_JSON\": '{\"datalad.foo\": \"val\"}',",
                                "type": "added"
                            },
                            {
                                "content": "                     \"DATALAD_FOO\": \"val-direct\"}):",
                                "type": "context"
                            },
                            {
                                "content": "        cfg.reload()",
                                "type": "context"
                            },
                            {
                                "content": "        assert_equal(cfg[\"datalad.foo\"], \"val-direct\")",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "    # JSON decode errors don't lead to crash.",
                                "type": "context"
                            },
                            {
                                "content": "    with patch.dict(\"os.environ\",",
                                "type": "context"
                            },
                            {
                                "content": "                    {\"DATALAD_CONFIG_OVERRIDES\": '{'}):",
                                "type": "deleted"
                            },
                            {
                                "content": "                    {\"DATALAD_CONFIG_OVERRIDES_JSON\": '{'}):",
                                "type": "added"
                            },
                            {
                                "content": "        with swallow_logs(logging.WARNING) as cml:",
                                "type": "context"
                            },
                            {
                                "content": "            cfg.reload()",
                                "type": "context"
                            },
                            {
                                "content": "        assert_in(\"Failed to load DATALAD_CONFIG_OVERRIDE\", cml.out)",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [
                        {
                            "file_name": "datalad/tests/test_config.py",
                            "type": "function",
                            "name": "test_from_env_overrides",
                            "content": "def test_from_env_overrides():\n    cfg = ConfigManager()\n    assert_not_in(\"datalad.FoO\", cfg)\n\n    # Some details, like case and underscores, cannot be handled by the direct\n    # environment variable mapping.\n    with patch.dict(\"os.environ\",\n                    {\"DATALAD_FOO\": \"val\"}):\n        cfg.reload()\n        assert_not_in(\"datalad.FoO\", cfg)\n        assert_equal(cfg[\"datalad.foo\"], \"val\")\n\n    # But they can be handled via DATALAD_CONFIG_OVERRIDES.\n    with patch.dict(\"os.environ\",\n                    {\"DATALAD_CONFIG_OVERRIDES\": '{\"datalad.FoO\": \"val\"}'}):\n        cfg.reload()\n        assert_equal(cfg[\"datalad.FoO\"], \"val\")\n\n    # DATALAD_CONFIG_OVERRIDES isn't limited to datalad variables.\n    with patch.dict(\"os.environ\",\n                    {\"DATALAD_CONFIG_OVERRIDES\": '{\"a.b.c\": \"val\"}'}):\n        cfg.reload()\n        assert_equal(cfg[\"a.b.c\"], \"val\")\n\n    # Explicitly provided DATALAD_ variables take precedence over those in\n    # DATALAD_CONFIG_OVERRIDES.\n    with patch.dict(\"os.environ\",\n                    {\"DATALAD_CONFIG_OVERRIDES\": '{\"datalad.foo\": \"val\"}',\n                     \"DATALAD_FOO\": \"val-direct\"}):\n        cfg.reload()\n        assert_equal(cfg[\"datalad.foo\"], \"val-direct\")\n\n    # JSON decode errors don't lead to crash.\n    with patch.dict(\"os.environ\",\n                    {\"DATALAD_CONFIG_OVERRIDES\": '{'}):\n        with swallow_logs(logging.WARNING) as cml:\n            cfg.reload()\n        assert_in(\"Failed to load DATALAD_CONFIG_OVERRIDE\", cml.out)",
                            "ast_type_str": "FunctionDef",
                            "start_lineno": 364,
                            "end_lineno": 401,
                            "parent": null
                        }
                    ],
                    "added_entities": [],
                    "deleted_entities": []
                },
                {
                    "descriptor": {
                        "old_range": {
                            "start": 640,
                            "length": 4
                        },
                        "new_range": {
                            "start": 640,
                            "length": 3
                        },
                        "section": "def test_external_modification(path):"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "    runner.run(['git', 'config', '--local', '--replace-all', key, '11'])",
                                "type": "context"
                            },
                            {
                                "content": "    config.reload()",
                                "type": "context"
                            },
                            {
                                "content": "    assert_equal(config[key], '11')",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "deleted"
                            }
                        ]
                    },
                    "modified_entities": [],
                    "added_entities": [],
                    "deleted_entities": []
                }
            ]
        },
        {
            "old_file_content": ".. _configuration:\n\nConfiguration\n*************\n\nDataLad uses the same configuration mechanism and syntax as Git itself.\nConsequently, datalad can be configured using the :command:`git config`\ncommand. Both a *global* user configuration (typically at\n:file:`~/.gitconfig`), and a *local* repository-specific configuration\n(:file:`.git/config`) are inspected.\n\nIn addition, datalad supports a persistent dataset-specific configuration.\nThis configuration is stored at :file:`.datalad/config` in any dataset.  As it\nis part of a dataset, settings stored there will also be in effect for any\nconsumer of such a dataset. Both *global* and *local* settings on a particular\nmachine always override configuration shipped with a dataset.\n\nAll datalad-specific configuration variables are prefixed with ``datalad.``.\n\nIt is possible to override or amend the configuration using environment\nvariables. Any variable with a name that starts with ``DATALAD_`` will\nbe available as the corresponding ``datalad.`` configuration variable,\nreplacing any ``__`` (two underscores) with a hyphen, then any ``_``\n(single underscore) with a dot, and finally converting all letters to\nlower case. Values from environment variables take precedence over\nconfiguration file settings.\n\nIn addition, the ``DATALAD_CONFIG_OVERRIDES`` environment variable can\nbe set to a JSON record with configuration values.  This is\nparticularly useful for options that aren't accessible through the\nnaming scheme described above (e.g., an option name that includes an\nunderscore).\n\nThe following sections provide a (non-exhaustive) list of settings honored\nby datalad. They are categorized according to the scope they are typically\nassociated with.\n\n\nGlobal user configuration\n=========================\n\n.. include:: generated/cfginfo/global.rst.in\n\nLocal repository configuration\n==============================\n\n.. include:: generated/cfginfo/local.rst.in\n\nSticky dataset configuration\n=============================\n\n.. include:: generated/cfginfo/dataset.rst.in\n\nMiscellaneous configuration\n===========================\n\n.. include:: generated/cfginfo/misc.rst.in\n",
            "new_file_content": ".. _configuration:\n\nConfiguration\n*************\n\nDataLad uses the same configuration mechanism and syntax as Git itself.\nConsequently, datalad can be configured using the :command:`git config`\ncommand. Both a *global* user configuration (typically at\n:file:`~/.gitconfig`), and a *local* repository-specific configuration\n(:file:`.git/config`) are inspected.\n\nIn addition, datalad supports a persistent dataset-specific configuration.\nThis configuration is stored at :file:`.datalad/config` in any dataset.  As it\nis part of a dataset, settings stored there will also be in effect for any\nconsumer of such a dataset. Both *global* and *local* settings on a particular\nmachine always override configuration shipped with a dataset.\n\nAll datalad-specific configuration variables are prefixed with ``datalad.``.\n\nIt is possible to override or amend the configuration using environment\nvariables. Any variable with a name that starts with ``DATALAD_`` will\nbe available as the corresponding ``datalad.`` configuration variable,\nreplacing any ``__`` (two underscores) with a hyphen, then any ``_``\n(single underscore) with a dot, and finally converting all letters to\nlower case. Values from environment variables take precedence over\nconfiguration file settings.\n\nIn addition, the ``DATALAD_CONFIG_OVERRIDES_JSON`` environment variable can\nbe set to a JSON record with configuration values.  This is\nparticularly useful for options that aren't accessible through the\nnaming scheme described above (e.g., an option name that includes an\nunderscore).\n\nThe following sections provide a (non-exhaustive) list of settings honored\nby datalad. They are categorized according to the scope they are typically\nassociated with.\n\n\nGlobal user configuration\n=========================\n\n.. include:: generated/cfginfo/global.rst.in\n\nLocal repository configuration\n==============================\n\n.. include:: generated/cfginfo/local.rst.in\n\nSticky dataset configuration\n=============================\n\n.. include:: generated/cfginfo/dataset.rst.in\n\nMiscellaneous configuration\n===========================\n\n.. include:: generated/cfginfo/misc.rst.in\n",
            "header": {
                "file": {
                    "path": "docs/source/config.rst"
                },
                "misc_line": null
            },
            "index_line": {
                "old_commit_hash": "40f89b9f0",
                "new_commit_hash": "443d1e450",
                "mode": "100644"
            },
            "is_binary_file": false,
            "binary_line": null,
            "minus_file": {
                "path": "a/docs/source/config.rst"
            },
            "plus_file": {
                "path": "b/docs/source/config.rst"
            },
            "hunks": [
                {
                    "descriptor": {
                        "old_range": {
                            "start": 25,
                            "length": 7
                        },
                        "new_range": {
                            "start": 25,
                            "length": 7
                        },
                        "section": "replacing any ``__`` (two underscores) with a hyphen, then any ``_``"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "lower case. Values from environment variables take precedence over",
                                "type": "context"
                            },
                            {
                                "content": "configuration file settings.",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "In addition, the ``DATALAD_CONFIG_OVERRIDES`` environment variable can",
                                "type": "deleted"
                            },
                            {
                                "content": "In addition, the ``DATALAD_CONFIG_OVERRIDES_JSON`` environment variable can",
                                "type": "added"
                            },
                            {
                                "content": "be set to a JSON record with configuration values.  This is",
                                "type": "context"
                            },
                            {
                                "content": "particularly useful for options that aren't accessible through the",
                                "type": "context"
                            },
                            {
                                "content": "naming scheme described above (e.g., an option name that includes an",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [],
                    "added_entities": [],
                    "deleted_entities": []
                }
            ]
        }
    ],
    "old_commit_hash": "b3c3916449e13b8e3de4ecf4e0aee59b7fbc28f8^",
    "new_commit_hash": "b3c3916449e13b8e3de4ecf4e0aee59b7fbc28f8",
    "commit_message": "RF: Rename DATALAD_CONFIG_OVERRIDES to DATALAD_CONFIG_OVERRIDES_JSON\n\nThanks @mih for the suggestion.",
    "commit_date": "2021-03-23T10:56:21-04:00",
    "metadata": {}
}
{
    "repo_name": "datalad",
    "new_commit_hash": "b3c3916449e13b8e3de4ecf4e0aee59b7fbc28f8",
    "test_file_codes": [
        "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-; coding: utf-8 -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"test command datalad run-procedure\n\n\"\"\"\n\n__docformat__ = 'restructuredtext'\n\nimport os.path as op\nimport sys\nfrom unittest.mock import patch\n\nfrom datalad.cmd import (\n    WitlessRunner,\n    KillOutput,\n)\nfrom datalad.utils import (\n    chpwd,\n    quote_cmdlinearg,\n    swallow_outputs,\n)\nfrom datalad.tests.utils import (\n    OBSCURE_FILENAME,\n    assert_false,\n    assert_in_results,\n    assert_not_in_results,\n    assert_raises,\n    assert_repo_status,\n    assert_result_count,\n    assert_true,\n    eq_,\n    known_failure_windows,\n    ok_file_has_content,\n    on_windows,\n    patch_config,\n    skip_if,\n    with_tempfile,\n    with_tree,\n)\nfrom datalad.distribution.dataset import Dataset\nfrom datalad.support.exceptions import (\n    CommandError,\n    InsufficientArgumentsError,\n)\nfrom datalad.api import run_procedure\nfrom datalad import cfg as dl_cfg\n\n\n@with_tempfile(mkdir=True)\ndef test_invalid_call(path):\n    with chpwd(path):\n        # ^ Change directory so that we don't fail with an\n        # InvalidGitRepositoryError if the test is executed from a git\n        # worktree.\n\n        # needs spec or discover\n        assert_raises(InsufficientArgumentsError, run_procedure)\n        res = run_procedure('unknown', on_failure='ignore')\n        assert_true(len(res) == 1)\n        assert_in_results(res, status=\"impossible\")\n\n\n@known_failure_windows  #FIXME\n@with_tree(tree={'README.md': 'dirty'})\ndef test_dirty(path):\n    ds = Dataset(path).create(force=True)\n    # must fail, because README.md is to be modified, but already dirty\n    assert_raises(CommandError, ds.run_procedure, 'cfg_yoda')\n    # make sure that was the issue\n    # save to git explicitly to keep the test simple and avoid unlocking...\n    ds.save('README.md', to_git=True)\n    ds.run_procedure('cfg_yoda')\n    assert_repo_status(ds.path)\n\n\n@skip_if(cond=on_windows and dl_cfg.obtain(\"datalad.repo.version\") < 6)\n@with_tree(tree={\n    'code': {'datalad_test_proc.py': \"\"\"\\\nimport sys\nimport os.path as op\nfrom datalad.api import save, Dataset\n\nwith open(op.join(sys.argv[1], 'fromproc.txt'), 'w') as f:\n    f.write('hello\\\\n')\nsave(dataset=Dataset(sys.argv[1]), path='fromproc.txt')\n\"\"\"},\n    'cfg_yoda.sh': \"\"\"\\\n#!/bin/bash\necho \"I am never to be ran but could be discovered\"\nexit 1\n\"\"\"})\n@with_tempfile\ndef test_procedure_discovery(path, super_path):\n    with chpwd(path):\n        # ^ Change directory so that we don't fail with an\n        # InvalidGitRepositoryError if the test is executed from a git\n        # worktree.\n        ps = run_procedure(discover=True)\n        # there are a few procedures coming with datalad, needs to find them\n        assert_true(len(ps) > 2)\n        # we get essential properties\n        _check_procedure_properties(ps)\n\n    # set up dataset with registered procedure (c&p from test_basics):\n    ds = Dataset(path).create(force=True)\n    # extra check: must not pick up cfg_yoda.sh in top directory\n    ds.run_procedure('cfg_yoda')\n\n    # path to a procedure which is not under any \"standard\" location but\n    # present in the dataset\n    code_dir_procedure_path = op.join(ds.path, 'code', 'datalad_test_proc.py')\n    top_dir_procedure_path = op.join(ds.path, 'cfg_yoda.sh')\n\n    # run discovery on the dataset:\n    ps = ds.run_procedure(discover=True)\n    # it should not be found magically by default\n    assert_not_in_results(ps, path=code_dir_procedure_path)\n    assert_not_in_results(ps, path=top_dir_procedure_path)\n\n    with patch_config({'datalad.locations.extra-procedures': op.join(ds.path, 'code')}):\n        # run discovery on the dataset:\n        ps = ds.run_procedure(discover=True)\n        # still needs to find procedures coming with datalad\n        assert_true(len(ps) > 3)\n        # and procedure under the path we specified\n        assert_result_count(ps, 1, path=code_dir_procedure_path)\n        assert_not_in_results(ps, path=top_dir_procedure_path)\n\n    # multiple extra locations\n    with patch_config({'datalad.locations.extra-procedures': [op.join(ds.path, 'code'), ds.path]}):\n        # run discovery on the dataset:\n        ps = ds.run_procedure(discover=True)\n        # still needs to find procedures coming with datalad\n        assert_true(len(ps) > 4)\n        # and procedure under the path we specified\n        assert_result_count(ps, 1, path=code_dir_procedure_path)\n        assert_result_count(ps, 1, path=top_dir_procedure_path)\n\n    # configure dataset to look for procedures in its code folder\n    ds.config.add(\n        'datalad.locations.dataset-procedures',\n        'code',\n        where='dataset')\n    ds.save(op.join('.datalad', 'config'))\n\n    # run discovery on the dataset:\n    ps = ds.run_procedure(discover=True)\n\n    # still needs to find procedures coming with datalad\n    assert_true(len(ps) > 2)\n    # we get three essential properties\n    _check_procedure_properties(ps)\n    # dataset's procedure needs to be in the results\n    # and only a single one\n    assert_result_count(ps, 1, path=code_dir_procedure_path)\n    # a subdir shouldn't be considered a procedure just because it's \"executable\"\n    assert_not_in_results(ps, path=op.join(ds.path, 'code', 'testdir'))\n\n    # make it a subdataset and try again:\n    # first we need to save the beast to make install work\n    ds.save()\n    super = Dataset(super_path).create()\n    super.install('sub', source=ds.path)\n\n    ps = super.run_procedure(discover=True)\n    # still needs to find procedures coming with datalad\n    assert_true(len(ps) > 2)\n    _check_procedure_properties(ps)\n    # dataset's procedure needs to be in the results\n    assert_in_results(ps, path=op.join(super.path, 'sub', 'code',\n                                       'datalad_test_proc.py'))\n\n    if not on_windows:  # no symlinks\n        import os\n        # create a procedure which is a broken symlink, but recognizable as a\n        # python script:\n        os.symlink(op.join(super.path, 'sub', 'not_existent'),\n                   op.join(super.path, 'sub', 'code', 'broken_link_proc.py'))\n        # broken symlink at procedure location, but we can't tell, whether it is\n        # an actual procedure without any guess on how to execute it:\n        os.symlink(op.join(super.path, 'sub', 'not_existent'),\n                   op.join(super.path, 'sub', 'code', 'unknwon_broken_link'))\n\n        ps = super.run_procedure(discover=True)\n        # still needs to find procedures coming with datalad and the dataset\n        # procedure registered before\n        assert_true(len(ps) > 3)\n        assert_in_results(ps, path=op.join(super.path, 'sub', 'code',\n                                           'broken_link_proc.py'),\n                          state='absent')\n        assert_in_results(\n            ps,\n            path=op.join(super.path, 'sub', 'code',\n                         'unknwon_broken_link'),\n            state='absent')\n\n\ndef _check_procedure_properties(ps):\n    \"\"\"a common check that we get three essential properties\"\"\"\n    eq_(\n        sum(['procedure_type' in p and\n             'procedure_callfmt' in p and\n             'path' in p\n             for p in ps]),\n        len(ps))\n\n\n@skip_if(cond=on_windows and dl_cfg.obtain(\"datalad.repo.version\") < 6)\n@with_tree(tree={\n    'code': {'datalad_test_proc.py': \"\"\"\\\nimport sys\nimport os.path as op\nfrom datalad.api import save, Dataset\n\nwith open(op.join(sys.argv[1], 'fromproc.txt'), 'w') as f:\n    f.write('{}\\\\n'.format(sys.argv[2]))\nsave(dataset=Dataset(sys.argv[1]), path='fromproc.txt')\n\"\"\"}})\ndef test_configs(path):\n\n    # set up dataset with registered procedure (c&p from test_basics):\n    ds = Dataset(path).create(force=True)\n    ds.run_procedure('cfg_yoda')\n    # configure dataset to look for procedures in its code folder\n    ds.config.add(\n        'datalad.locations.dataset-procedures',\n        'code',\n        where='dataset')\n\n    # 1. run procedure based on execution guessing by run_procedure:\n    ds.run_procedure(spec=['datalad_test_proc', 'some_arg'])\n    # look for traces\n    ok_file_has_content(op.join(ds.path, 'fromproc.txt'), 'some_arg\\n')\n\n    # 2. now configure specific call format including usage of substitution config\n    # for run:\n    ds.config.add(\n        'datalad.procedures.datalad_test_proc.call-format',\n        u'%s {script} {ds} {{mysub}} {args}' % quote_cmdlinearg(sys.executable),\n        where='dataset'\n    )\n    ds.config.add(\n        'datalad.run.substitutions.mysub',\n        'dataset-call-config',\n        where='dataset'\n    )\n    # TODO: Should we allow for --inputs/--outputs arguments for run_procedure\n    #       (to be passed into run)?\n    ds.unlock(\"fromproc.txt\")\n    # run again:\n    ds.run_procedure(spec=['datalad_test_proc', 'some_arg'])\n    # look for traces\n    ok_file_has_content(op.join(ds.path, 'fromproc.txt'), 'dataset-call-config\\n')\n\n    # 3. have a conflicting config at user-level, which should override the\n    # config on dataset level:\n    ds.config.add(\n        'datalad.procedures.datalad_test_proc.call-format',\n        u'%s {script} {ds} local {args}' % quote_cmdlinearg(sys.executable),\n        where='local'\n    )\n    ds.unlock(\"fromproc.txt\")\n    # run again:\n    ds.run_procedure(spec=['datalad_test_proc', 'some_arg'])\n    # look for traces\n    ok_file_has_content(op.join(ds.path, 'fromproc.txt'), 'local\\n')\n\n    # 4. get configured help message:\n    r = ds.run_procedure('datalad_test_proc', help_proc=True,\n                         on_failure='ignore')\n    assert_true(len(r) == 1)\n    assert_in_results(r, status=\"impossible\")\n\n    ds.config.add(\n        'datalad.procedures.datalad_test_proc.help',\n        \"This is a help message\",\n        where='dataset'\n    )\n\n    r = ds.run_procedure('datalad_test_proc', help_proc=True)\n    assert_true(len(r) == 1)\n    assert_in_results(r, message=\"This is a help message\", status='ok')\n\n\n@known_failure_windows\n@with_tree(tree={\n    'code': {'datalad_test_proc.py': \"\"\"\\\nimport sys\nimport os.path as op\nfrom datalad.api import save, Dataset\n\nwith open(op.join(sys.argv[1], sys.argv[2]), 'w') as f:\n    f.write('hello\\\\n')\nsave(dataset=Dataset(sys.argv[1]), path=sys.argv[2])\n\"\"\"}})\ndef test_spaces(path):\n    \"\"\"\n    Test whether args with spaces are correctly parsed.\n    \"\"\"\n    ds = Dataset(path).create(force=True)\n    ds.run_procedure('cfg_yoda')\n    # configure dataset to look for procedures in its code folder\n    ds.config.add(\n        'datalad.locations.dataset-procedures',\n        'code',\n        where='dataset')\n    # 1. run procedure based on execution guessing by run_procedure:\n    ds.run_procedure(spec=['datalad_test_proc', 'with spaces', 'unrelated'])\n    # check whether file has name with spaces\n    ok_file_has_content(op.join(ds.path, 'with spaces'), 'hello\\n')\n\n\n@known_failure_windows\n@with_tree(tree={OBSCURE_FILENAME:\n                 {\"code\": {\"just2args.py\": \"\"\"\nimport sys\nprint(sys.argv)\n# script, dataset, and two others\nassert len(sys.argv) == 4\n\"\"\"}}})\ndef test_quoting(path):\n    ds = Dataset(op.join(path, OBSCURE_FILENAME)).create(force=True)\n    # Our custom procedure fails if it receives anything other than two\n    # procedure arguments (so the script itself receives 3). Check a few cases\n    # from the Python API and CLI.\n    ds.config.add(\"datalad.locations.dataset-procedures\", \"code\",\n                  where=\"dataset\")\n    with swallow_outputs():\n        ds.run_procedure(spec=[\"just2args\", \"with ' sing\", 'with \" doub'])\n        with assert_raises(CommandError):\n            ds.run_procedure(spec=[\"just2args\", \"still-one arg\"])\n\n        runner = WitlessRunner(cwd=ds.path)\n        runner.run(\n            \"datalad run-procedure just2args \\\"with ' sing\\\" 'with \\\" doub'\",\n            protocol=KillOutput)\n        with assert_raises(CommandError):\n            runner.run(\n                \"datalad run-procedure just2args 'still-one arg'\",\n                protocol=KillOutput)\n\n\n@with_tree(tree={\n    # \"TEXT\" ones\n    'empty': '',  # we have special rule to treat empty ones as text\n    # check various structured files - libmagic might change its decisions which\n    # can effect git-annex. https://github.com/datalad/datalad/issues/3361\n    'JSON': \"\"\"\\\n{\n    \"name\": \"John Smith\",\n    \"age\": 33\n}\n\"\"\",\n    'YAML': \"\"\"\\\n--- # The Smiths\n- {name: John Smith, age: 33}\n- name: Mary Smith\n  age: 27\n\"\"\",\n    'MARKDOWN': \"\"\"\\\n# Title\n\n## Section1\n\nWhen the earth was flat\n\n## Section2\n\"\"\",\n    # BINARY ones\n    '0blob': '\\x00',\n    'emptyline': '\\n',  # libmagic: \"binary\" \"application/octet-stream\"\n})\ndef test_text2git(path):\n    # Test if files being correctly annexed in a ds configured with text2git.\n    TEXT_FILES = ('JSON', 'YAML', 'MARKDOWN', 'empty')\n    BINARY_FILES = ('0blob', 'emptyline')\n\n    ds = Dataset(path).create(force=True)\n    ds.run_procedure('cfg_text2git')\n    ds.save(path=TEXT_FILES + BINARY_FILES, message=\"added all files\")\n    assert_repo_status(ds.path)\n\n    # check that text files are not annexed\n    for f in TEXT_FILES:\n        assert_false(ds.repo.is_under_annex(f))\n    # and trivial binaries - annexed\n    for f in BINARY_FILES:\n        assert_true(ds.repo.is_under_annex(f))\n\n\n@with_tree(tree={\".datalad\": {\"procedures\": {\"print_args\": \"\"\"\nimport sys\nprint(sys.argv)\n\"\"\"}}})\ndef test_name_with_underscore(path):\n    ds = Dataset(path).create(force=True)\n\n    # Procedure name with underscore can't be reached directly with a DATALAD_\n    # environment variable.\n    with patch.dict(\"os.environ\",\n                    {\"DATALAD_PROCEDURES_PRINT_ARGS_CALL__FORMAT\":\n                     'python {script}'}):\n        with assert_raises(ValueError):\n            ds.run_procedure(spec=[\"print_args\"])\n\n    # But it can be set via DATALAD_CONFIG_OVERRIDES_JSON.\n    with patch.dict(\"os.environ\",\n                    {\"DATALAD_CONFIG_OVERRIDES_JSON\":\n                     '{\"datalad.procedures.print_args.call-format\": '\n                     '\"python {script}\"}'}):\n        ds.config.reload()\n        ds.run_procedure(spec=[\"print_args\"])\n",
        "# ex: set sts=4 ts=4 sw=4 noet:\n# -*- coding: utf-8 -*-\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"\n\n\"\"\"\n\nimport logging\nimport os\nfrom os.path import exists\nfrom os.path import join as opj\n\nfrom unittest.mock import patch\nfrom datalad.tests.utils import (\n    assert_equal,\n    assert_false,\n    assert_in,\n    assert_not_equal,\n    assert_not_in,\n    assert_raises,\n    assert_true,\n    chpwd,\n    ok_file_has_content,\n    with_tempfile,\n    with_testsui,\n    with_tree,\n)\nfrom datalad.utils import (\n    get_home_envvars,\n    swallow_logs,\n    Path\n)\n\nfrom datalad.distribution.dataset import Dataset\nfrom datalad.api import create\nfrom datalad.config import (\n    ConfigManager,\n    rewrite_url,\n    write_config_section,\n)\nfrom datalad.cmd import CommandError\n\nfrom datalad.support.gitrepo import GitRepo\nfrom datalad import cfg as dl_cfg\n\n\n# XXX tabs are intentional (part of the format)!\n# XXX put back! confuses pep8\n_config_file_content = \"\"\"\\\n[something]\nuser = name=Jane Doe\nuser = email=jd@example.com\nnovalue\nempty =\nmyint = 3\n\n[onemore \"complicated の beast with.dot\"]\nfindme = 5.0\n\"\"\"\n\n_dataset_config_template = {\n    'ds': {\n        '.datalad': {\n            'config': _config_file_content}}}\n\n\n@with_tree(tree=_dataset_config_template)\n@with_tempfile(mkdir=True)\ndef test_something(path, new_home):\n    # will refuse to work on dataset without a dataset\n    assert_raises(ValueError, ConfigManager, source='dataset')\n    # now read the example config\n    cfg = ConfigManager(GitRepo(opj(path, 'ds'), create=True), source='dataset')\n    assert_equal(len(cfg), 5)\n    assert_in('something.user', cfg)\n    # multi-value\n    assert_equal(len(cfg['something.user']), 2)\n    assert_equal(cfg['something.user'], ('name=Jane Doe', 'email=jd@example.com'))\n\n    assert_true(cfg.has_section('something'))\n    assert_false(cfg.has_section('somethingelse'))\n    assert_equal(sorted(cfg.sections()),\n                 [u'onemore.complicated の beast with.dot', 'something'])\n    assert_true(cfg.has_option('something', 'user'))\n    assert_false(cfg.has_option('something', 'us?er'))\n    assert_false(cfg.has_option('some?thing', 'user'))\n    assert_equal(sorted(cfg.options('something')), ['empty', 'myint', 'novalue', 'user'])\n    assert_equal(cfg.options(u'onemore.complicated の beast with.dot'), ['findme'])\n\n    assert_equal(\n        sorted(cfg.items()),\n        [(u'onemore.complicated の beast with.dot.findme', '5.0'),\n         ('something.empty', ''),\n         ('something.myint', '3'),\n         ('something.novalue', None),\n         ('something.user', ('name=Jane Doe', 'email=jd@example.com'))])\n    assert_equal(\n        sorted(cfg.items('something')),\n        [('something.empty', ''),\n         ('something.myint', '3'),\n         ('something.novalue', None),\n         ('something.user', ('name=Jane Doe', 'email=jd@example.com'))])\n\n    # by default get last value only\n    assert_equal(\n        cfg.get('something.user'), 'email=jd@example.com')\n    # but can get all values\n    assert_equal(\n        cfg.get('something.user', get_all=True),\n        ('name=Jane Doe', 'email=jd@example.com'))\n    assert_raises(KeyError, cfg.__getitem__, 'somedthing.user')\n    assert_equal(cfg.getfloat(u'onemore.complicated の beast with.dot', 'findme'), 5.0)\n    assert_equal(cfg.getint('something', 'myint'), 3)\n    assert_equal(cfg.getbool('something', 'myint'), True)\n    # git demands a key without value at all to be used as a flag, thus True\n    assert_equal(cfg.getbool('something', 'novalue'), True)\n    assert_equal(cfg.get('something.novalue'), None)\n    # empty value is False\n    assert_equal(cfg.getbool('something', 'empty'), False)\n    assert_equal(cfg.get('something.empty'), '')\n    assert_equal(cfg.getbool('doesnot', 'exist', default=True), True)\n    assert_raises(TypeError, cfg.getbool, 'something', 'user')\n\n    # gitpython-style access\n    assert_equal(cfg.get('something.myint'), cfg.get_value('something', 'myint'))\n    assert_equal(cfg.get_value('doesnot', 'exist', default='oohaaa'), 'oohaaa')\n    # weired, but that is how it is\n    assert_raises(KeyError, cfg.get_value, 'doesnot', 'exist', default=None)\n\n    # modification follows\n    cfg.add('something.new', 'の')\n    assert_equal(cfg.get('something.new'), u'の')\n    # sections are added on demand\n    cfg.add('unheard.of', 'fame')\n    assert_true(cfg.has_section('unheard.of'))\n    comp = cfg.items('something')\n    cfg.rename_section('something', 'this')\n    assert_true(cfg.has_section('this'))\n    assert_false(cfg.has_section('something'))\n    # direct comparision would fail, because of section prefix\n    assert_equal(len(cfg.items('this')), len(comp))\n    # fail if no such section\n    with swallow_logs():\n        assert_raises(CommandError, cfg.rename_section, 'nothere', 'irrelevant')\n    assert_true(cfg.has_option('this', 'myint'))\n    cfg.unset('this.myint')\n    assert_false(cfg.has_option('this', 'myint'))\n\n    # batch a changes\n    cfg.add('mike.wants.to', 'know', reload=False)\n    assert_false('mike.wants.to' in cfg)\n    cfg.add('mike.wants.to', 'eat')\n    assert_true('mike.wants.to' in cfg)\n    assert_equal(len(cfg['mike.wants.to']), 2)\n\n    # set a new one:\n    cfg.set('mike.should.have', 'known')\n    assert_in('mike.should.have', cfg)\n    assert_equal(cfg['mike.should.have'], 'known')\n    # set an existing one:\n    cfg.set('mike.should.have', 'known better')\n    assert_equal(cfg['mike.should.have'], 'known better')\n    # set, while there are several matching ones already:\n    cfg.add('mike.should.have', 'a meal')\n    assert_equal(len(cfg['mike.should.have']), 2)\n    # raises with force=False\n    assert_raises(CommandError,\n                  cfg.set, 'mike.should.have', 'a beer', force=False)\n    assert_equal(len(cfg['mike.should.have']), 2)\n    # replaces all matching ones with force=True\n    cfg.set('mike.should.have', 'a beer', force=True)\n    assert_equal(cfg['mike.should.have'], 'a beer')\n\n    # fails unknown location\n    assert_raises(ValueError, cfg.add, 'somesuch', 'shit', where='umpalumpa')\n\n    # very carefully test non-local config\n    # so carefully that even in case of bad weather Yarik doesn't find some\n    # lame datalad unittest sections in his precious ~/.gitconfig\n    env = get_home_envvars(new_home)\n    with patch.dict('os.environ',\n                    dict(get_home_envvars(new_home), DATALAD_SNEAKY_ADDITION='ignore')):\n        global_gitconfig = opj(new_home, '.gitconfig')\n        assert(not exists(global_gitconfig))\n        globalcfg = ConfigManager()\n        assert_not_in('datalad.unittest.youcan', globalcfg)\n        assert_in('datalad.sneaky.addition', globalcfg)\n        cfg.add('datalad.unittest.youcan', 'removeme', where='global')\n        assert(exists(global_gitconfig))\n        # it did not go into the dataset's config!\n        assert_not_in('datalad.unittest.youcan', cfg)\n        # does not monitor additions!\n        globalcfg.reload(force=True)\n        assert_in('datalad.unittest.youcan', globalcfg)\n        with swallow_logs():\n            assert_raises(\n                CommandError,\n                globalcfg.unset,\n                'datalad.unittest.youcan',\n                where='local')\n        assert(globalcfg.has_section('datalad.unittest'))\n        globalcfg.unset('datalad.unittest.youcan', where='global')\n        # but after we unset the only value -- that section is no longer listed\n        assert (not globalcfg.has_section('datalad.unittest'))\n        assert_not_in('datalad.unittest.youcan', globalcfg)\n        ok_file_has_content(global_gitconfig, \"\")\n\n    cfg = ConfigManager(\n        Dataset(opj(path, 'ds')),\n        source='dataset',\n        overrides={'datalad.godgiven': True})\n    assert_equal(cfg.get('datalad.godgiven'), True)\n    # setter has no effect\n    cfg.set('datalad.godgiven', 'false')\n    assert_equal(cfg.get('datalad.godgiven'), True)\n\n\n@with_tree(tree={\n    'ds': {\n        '.datalad': {\n            'config': \"\"\"\\\n[crazy]\n    fa = !git remote | xargs -r -I REMOTE /bin/bash -c 'echo I: Fetching from REMOTE && git fetch --prune REMOTE && git fetch -t REMOTE' && [ -d .git/svn ] && bash -c 'echo I: Fetching from SVN && git svn fetch' || : && [ -e .gitmodules ] && bash -c 'echo I: Fetching submodules && git submodule foreach git fa' && [ -d .git/sd ] && bash -c 'echo I: Fetching bugs into sd && git-sd pull --all' || :\n    pa = !git paremotes | tr ' ' '\\\\n'  | xargs -r -l1 git push\n    pt = !git testremotes | tr ' ' '\\\\n'  | xargs -r -l1 -I R git push -f R master\n    ptdry = !git testremotes | tr ' ' '\\\\n'  | xargs -r -l1 -I R git push -f --dry-run R master\n    padry = !git paremotes | tr ' ' '\\\\n' | xargs -r -l1 git push --dry-run\n\"\"\"}}})\ndef test_crazy_cfg(path):\n    cfg = ConfigManager(GitRepo(opj(path, 'ds'), create=True), source='dataset')\n    assert_in('crazy.padry', cfg)\n    # make sure crazy config is not read when in local mode\n    cfg = ConfigManager(Dataset(opj(path, 'ds')), source='local')\n    assert_not_in('crazy.padry', cfg)\n    # it will make it in in 'any' mode though\n    cfg = ConfigManager(Dataset(opj(path, 'ds')), source='any')\n    assert_in('crazy.padry', cfg)\n    # typos in the source mode arg will not have silent side-effects\n    assert_raises(\n        ValueError, ConfigManager, Dataset(opj(path, 'ds')), source='locale')\n\n\n@with_tempfile\ndef test_obtain(path):\n    ds = create(path)\n    cfg = ConfigManager(ds)\n    dummy = 'datalad.test.dummy'\n    # we know nothing and we don't know how to ask\n    assert_raises(RuntimeError, cfg.obtain, dummy)\n    # can report known ones\n    cfg.add(dummy, '5.3')\n    assert_equal(cfg.obtain(dummy), '5.3')\n    # better type\n    assert_equal(cfg.obtain(dummy, valtype=float), 5.3)\n    # don't hide type issues, float doesn't become an int magically\n    assert_raises(ValueError, cfg.obtain, dummy, valtype=int)\n    # inject some prior knowledge\n    from datalad.interface.common_cfg import definitions as cfg_defs\n    cfg_defs[dummy] = dict(type=float)\n    # no we don't need to specify a type anymore\n    assert_equal(cfg.obtain(dummy), 5.3)\n    # but if we remove the value from the config, all magic is gone\n    cfg.unset(dummy)\n    # we know nothing and we don't know how to ask\n    assert_raises(RuntimeError, cfg.obtain, dummy)\n\n    #\n    # test actual interaction\n    #\n    @with_testsui()\n    def ask():\n        # fail on unkown dialog type\n        assert_raises(ValueError, cfg.obtain, dummy, dialog_type='Rorschach_test')\n    ask()\n\n    # ask nicely, and get a value of proper type using the preconfiguration\n    @with_testsui(responses='5.3')\n    def ask():\n        assert_equal(\n            cfg.obtain(dummy, dialog_type='question', text='Tell me'), 5.3)\n    ask()\n\n    # preconfigure even more, to get the most compact call\n    cfg_defs[dummy]['ui'] = ('question', dict(text='tell me', title='Gretchen Frage'))\n\n    @with_testsui(responses='5.3')\n    def ask():\n        assert_equal(cfg.obtain(dummy), 5.3)\n    ask()\n\n    @with_testsui(responses='murks')\n    def ask():\n        assert_raises(ValueError, cfg.obtain, dummy)\n    ask()\n\n    # fail to store when destination is not specified, will not even ask\n    @with_testsui()\n    def ask():\n        assert_raises(ValueError, cfg.obtain, dummy, store=True)\n    ask()\n\n    # but we can preconfigure it\n    cfg_defs[dummy]['destination'] = 'broken'\n\n    @with_testsui(responses='5.3')\n    def ask():\n        assert_raises(ValueError, cfg.obtain, dummy, store=True)\n    ask()\n\n    # fixup destination\n    cfg_defs[dummy]['destination'] = 'dataset'\n\n    @with_testsui(responses='5.3')\n    def ask():\n        assert_equal(cfg.obtain(dummy, store=True), 5.3)\n    ask()\n\n    # now it won't have to ask again\n    @with_testsui()\n    def ask():\n        assert_equal(cfg.obtain(dummy), 5.3)\n    ask()\n\n    # wipe it out again\n    cfg.unset(dummy)\n    assert_not_in(dummy, cfg)\n\n    # XXX cannot figure out how I can simulate a simple <Enter>\n    ## respond with accepting the default\n    #@with_testsui(responses=...)\n    #def ask():\n    #    assert_equal(cfg.obtain(dummy, default=5.3), 5.3)\n    #ask()\n\n\ndef test_from_env():\n    cfg = ConfigManager()\n    assert_not_in('datalad.crazy.cfg', cfg)\n    with patch.dict('os.environ',\n                    {'DATALAD_CRAZY_CFG': 'impossibletoguess'}):\n        cfg.reload()\n        assert_in('datalad.crazy.cfg', cfg)\n        assert_equal(cfg['datalad.crazy.cfg'], 'impossibletoguess')\n        # not in dataset-only mode\n        cfg = ConfigManager(Dataset('nowhere'), source='dataset')\n        assert_not_in('datalad.crazy.cfg', cfg)\n    # check env trumps override\n    cfg = ConfigManager()\n    assert_not_in('datalad.crazy.override', cfg)\n    cfg.set('datalad.crazy.override', 'fromoverride', where='override')\n    cfg.reload()\n    assert_equal(cfg['datalad.crazy.override'], 'fromoverride')\n    with patch.dict('os.environ',\n                    {'DATALAD_CRAZY_OVERRIDE': 'fromenv'}):\n        cfg.reload()\n        assert_equal(cfg['datalad.crazy.override'], 'fromenv')\n\n\ndef test_from_env_overrides():\n    cfg = ConfigManager()\n    assert_not_in(\"datalad.FoO\", cfg)\n\n    # Some details, like case and underscores, cannot be handled by the direct\n    # environment variable mapping.\n    with patch.dict(\"os.environ\",\n                    {\"DATALAD_FOO\": \"val\"}):\n        cfg.reload()\n        assert_not_in(\"datalad.FoO\", cfg)\n        assert_equal(cfg[\"datalad.foo\"], \"val\")\n\n    # But they can be handled via DATALAD_CONFIG_OVERRIDES_JSON.\n    with patch.dict(\"os.environ\",\n                    {\"DATALAD_CONFIG_OVERRIDES_JSON\": '{\"datalad.FoO\": \"val\"}'}):\n        cfg.reload()\n        assert_equal(cfg[\"datalad.FoO\"], \"val\")\n\n    # DATALAD_CONFIG_OVERRIDES_JSON isn't limited to datalad variables.\n    with patch.dict(\"os.environ\",\n                    {\"DATALAD_CONFIG_OVERRIDES_JSON\": '{\"a.b.c\": \"val\"}'}):\n        cfg.reload()\n        assert_equal(cfg[\"a.b.c\"], \"val\")\n\n    # Explicitly provided DATALAD_ variables take precedence over those in\n    # DATALAD_CONFIG_OVERRIDES_JSON.\n    with patch.dict(\"os.environ\",\n                    {\"DATALAD_CONFIG_OVERRIDES_JSON\": '{\"datalad.foo\": \"val\"}',\n                     \"DATALAD_FOO\": \"val-direct\"}):\n        cfg.reload()\n        assert_equal(cfg[\"datalad.foo\"], \"val-direct\")\n\n    # JSON decode errors don't lead to crash.\n    with patch.dict(\"os.environ\",\n                    {\"DATALAD_CONFIG_OVERRIDES_JSON\": '{'}):\n        with swallow_logs(logging.WARNING) as cml:\n            cfg.reload()\n        assert_in(\"Failed to load DATALAD_CONFIG_OVERRIDE\", cml.out)\n\n\ndef test_overrides():\n    cfg = ConfigManager()\n    # any sensible (and also our CI) test environment(s) should have this\n    assert_in('user.name', cfg)\n    # set\n    cfg.set('user.name', 'myoverride', where='override')\n    assert_equal(cfg['user.name'], 'myoverride')\n    # unset just removes override, not entire config\n    cfg.unset('user.name', where='override')\n    assert_in('user.name', cfg)\n    assert_not_equal('user.name', 'myoverride')\n    # add\n    # there is no initial increment\n    cfg.add('user.name', 'myoverride', where='override')\n    assert_equal(cfg['user.name'], 'myoverride')\n    # same as with add, not a list\n    assert_equal(cfg['user.name'], 'myoverride')\n    # but then there is\n    cfg.add('user.name', 'myother', where='override')\n    assert_equal(cfg['user.name'], ['myoverride', 'myother'])\n    # rename\n    assert_not_in('ups.name', cfg)\n    cfg.rename_section('user', 'ups', where='override')\n    # original variable still there\n    assert_in('user.name', cfg)\n    # rename of override in effect\n    assert_equal(cfg['ups.name'], ['myoverride', 'myother'])\n    # remove entirely by section\n    cfg.remove_section('ups', where='override')\n    from datalad.utils import Path\n    assert_not_in(\n        'ups.name', cfg,\n        (cfg._stores,\n         cfg.overrides,\n    ))\n\n\ndef test_rewrite_url():\n    test_cases = (\n        # no match\n        ('unicorn', 'unicorn'),\n        # custom label replacement\n        ('example:datalad/datalad.git', 'git@example.com:datalad/datalad.git'),\n        # protocol enforcement\n        ('git://example.com/some', 'https://example.com/some'),\n        # multi-match\n        ('mylabel', 'ria+ssh://fully.qualified.com'),\n        ('myotherlabel', 'ria+ssh://fully.qualified.com'),\n        # conflicts, same label pointing to different URLs\n        ('conflict', 'conflict'),\n        # also conflicts, but hidden in a multi-value definition\n        ('conflict2', 'conflict2'),\n    )\n    cfg_in = {\n        # label rewrite\n        'git@example.com:': 'example:',\n        # protocol change\n        'https://example': 'git://example',\n        # multi-value\n        'ria+ssh://fully.qualified.com': ('mylabel', 'myotherlabel'),\n        # conflicting definitions\n        'http://host1': 'conflict',\n        'http://host2': 'conflict',\n        # hidden conflict\n        'http://host3': 'conflict2',\n        'http://host4': ('someokish', 'conflict2'),\n    }\n    cfg = {\n        'url.{}.insteadof'.format(k): v\n        for k, v in cfg_in.items()\n    }\n    for input, output in test_cases:\n        with swallow_logs(logging.WARNING) as msg:\n            assert_equal(rewrite_url(cfg, input), output)\n        if input.startswith('conflict'):\n            assert_in(\"Ignoring URL rewrite\", msg.out)\n\n\n# https://github.com/datalad/datalad/issues/4071\n@with_tempfile()\n@with_tempfile()\ndef test_no_leaks(path1, path2):\n    ds1 = Dataset(path1).create()\n    ds1.config.set('i.was.here', 'today', where='local')\n    assert_in('i.was.here', ds1.config.keys())\n    ds1.config.reload()\n    assert_in('i.was.here', ds1.config.keys())\n    # now we move into this one repo, and create another\n    # make sure that no config from ds1 leaks into ds2\n    with chpwd(path1):\n        ds2 = Dataset(path2)\n        assert_not_in('i.was.here', ds2.config.keys())\n        ds2.config.reload()\n        assert_not_in('i.was.here', ds2.config.keys())\n\n        ds2.create()\n        assert_not_in('i.was.here', ds2.config.keys())\n\n        # and that we do not track the wrong files\n        assert_not_in(ds1.pathobj / '.git' / 'config',\n                      ds2.config._stores['git']['files'])\n        assert_not_in(ds1.pathobj / '.datalad' / 'config',\n                      ds2.config._stores['dataset']['files'])\n        # these are the right ones\n        assert_in(ds2.pathobj / '.git' / 'config',\n                  ds2.config._stores['git']['files'])\n        assert_in(ds2.pathobj / '.datalad' / 'config',\n                  ds2.config._stores['dataset']['files'])\n\n\n@with_tempfile()\ndef test_no_local_write_if_no_dataset(path):\n    Dataset(path).create()\n    with chpwd(path):\n        cfg = ConfigManager()\n        with assert_raises(CommandError):\n            cfg.set('a.b.c', 'd', where='local')\n\n\n@with_tempfile\ndef test_dataset_local_mode(path):\n    ds = create(path)\n    # any sensible (and also our CI) test environment(s) should have this\n    assert_in('user.name', ds.config)\n    # from datalad.tests.datalad/config\n    assert_in('datalad.dataset.id', ds.config)\n    # from datalad.tests.git/config\n    assert_in('annex.version', ds.config)\n    # now check that dataset-local mode doesn't have the global piece\n    cfg = ConfigManager(ds, source='dataset-local')\n    assert_not_in('user.name', cfg)\n    assert_in('datalad.dataset.id', cfg)\n    assert_in('annex.version', cfg)\n\n\n# https://github.com/datalad/datalad/issues/4071\n@with_tempfile\ndef test_dataset_systemglobal_mode(path):\n    ds = create(path)\n    # any sensible (and also our CI) test environment(s) should have this\n    assert_in('user.name', ds.config)\n    # from datalad.tests.datalad/config\n    assert_in('datalad.dataset.id', ds.config)\n    # from datalad.tests.git/config\n    assert_in('annex.version', ds.config)\n    with chpwd(path):\n        # now check that no config from a random dataset at PWD is picked up\n        # if not dataset instance was provided\n        cfg = ConfigManager(dataset=None, source='any')\n        assert_in('user.name', cfg)\n        assert_not_in('datalad.dataset.id', cfg)\n        assert_not_in('annex.version', cfg)\n\n\ndef test_global_config():\n\n    # from within tests, global config should be read from faked $HOME (see\n    # setup_package)\n    glb_cfg_file = Path(os.path.expanduser('~')) / '.gitconfig'\n    assert any(glb_cfg_file.samefile(Path(p)) for p in dl_cfg._stores['git']['files'])\n    assert_equal(dl_cfg.get(\"user.name\"), \"DataLad Tester\")\n    assert_equal(dl_cfg.get(\"user.email\"), \"test@example.com\")\n\n\n@with_tempfile()\ndef test_bare(path):\n    # can we handle a bare repo?\n    gr = GitRepo(path, create=True, bare=True)\n    # do we read the correct local config?\n    assert_in(gr.pathobj / 'config', gr.config._stores['git']['files'])\n    # any sensible (and also our CI) test environment(s) should have this\n    assert_in('user.name', gr.config)\n    # not set something that wasn't there\n    obscure_key = 'sec.reallyobscurename!@@.key'\n    assert_not_in(obscure_key, gr.config)\n    # to the local config, which is easily accessible\n    gr.config.set(obscure_key, 'myvalue', where='local')\n    assert_equal(gr.config.get(obscure_key), 'myvalue')\n    # now make sure the config is where we think it is\n    assert_in(obscure_key.split('.')[1], (gr.pathobj / 'config').read_text())\n\n\n@with_tempfile()\ndef test_write_config_section(path):\n    # can we handle a bare repo?\n    gr = GitRepo(path, create=True, bare=True)\n\n    # test cases\n    # first 3 args are write_config_section() parameters\n    # 4th arg is a list with key/value pairs that should end up in a\n    # ConfigManager after a reload\n    testcfg = [\n        ('submodule', 'sub', dict(active='true', url='http://example.com'), [\n            ('submodule.sub.active', 'true'),\n            ('submodule.sub.url', 'http://example.com'),\n        ]),\n        ('submodule', 'sub\"quote', {\"a-b\": '\"quoted\"', 'c': 'with\"quote'}, [\n            ('submodule.sub\"quote.a-b', '\"quoted\"'),\n            ('submodule.sub\"quote.c', 'with\"quote'),\n        ]),\n        ('short', ' s p a c e ', {\"a123\": ' space all over '}, [\n            ('short. s p a c e .a123', ' space all over '),\n        ]),\n    ]\n\n    for tc in testcfg:\n        # using append mode to provoke potential interference by\n        # successive calls\n        with (gr.pathobj / 'config').open('a') as fobj:\n            write_config_section(fobj, tc[0], tc[1], tc[2])\n        gr.config.reload()\n        for testcase in tc[3]:\n            assert_in(testcase[0], gr.config)\n            assert_equal(testcase[1], gr.config[testcase[0]])\n\n\n@with_tempfile()\ndef test_external_modification(path):\n    from datalad.cmd import WitlessRunner as Runner\n    runner = Runner(cwd=path)\n    repo = GitRepo(path, create=True)\n    config = repo.config\n\n    key = 'sec.sub.key'\n    assert_not_in(key, config)\n    config.set(key, '1', where='local')\n    assert_equal(config[key], '1')\n\n    # we pick up the case where we modified so size changed\n    runner.run(['git', 'config', '--local', '--replace-all', key, '10'])\n    # unfortunately we do not react for .get unless reload. But here\n    # we will test if reload is correctly decides to reload without force\n    config.reload()\n    assert_equal(config[key], '10')\n\n    # and no size change\n    runner.run(['git', 'config', '--local', '--replace-all', key, '11'])\n    config.reload()\n    assert_equal(config[key], '11')\n",
        "import pytest\nimport tempfile\nimport os\n\n\n@pytest.fixture\ndef path():\n    with tempfile.TemporaryDirectory() as tmpdir:\n        yield tmpdir\n\n\n@pytest.fixture(autouse=True)\ndef setup_git_config():\n    with tempfile.TemporaryDirectory() as home:\n        old_home = os.environ.get(\"HOME\")\n        os.environ[\"HOME\"] = home\n        os.system('git config --global user.name \"DataLad Tester\"')\n        os.system('git config --global user.email \"test@example.com\"')\n        yield\n        if old_home:\n            os.environ[\"HOME\"] = old_home\n"
    ],
    "test_file_names": [
        "test_1.py",
        "test_2.py",
        "conftest.py"
    ],
    "setup_res_code": 0,
    "setup_res_stdout": "Starting Datalad installation attempts...\nVerifying installation...\nDatalad version: 0.14.0.dev65\n✅ Installation successful!\nSuccessfully installed Datalad using Python 3.9\n",
    "setup_res_stderr": "Using CPython 3.9.21\nCreating virtual environment at: .venv\nActivate with: source .venv/bin/activate\nResolved 25 packages in 108ms\nInstalled 25 packages in 5.22s\n + coverage==7.6.10\n + docopt==0.6.2\n + exceptiongroup==1.2.2\n + grabbit==0.2.6\n + importlib-resources==6.4.5\n + iniconfig==2.0.0\n + nibabel==5.3.2\n + num2words==0.5.14\n + numpy==2.0.2\n + packaging==24.2\n + pandas==2.2.3\n + patsy==1.0.1\n + pluggy==1.5.0\n + pybids==0.6.5\n + pytest==8.3.4\n + pytest-cov==6.0.0\n + python-dateutil==2.9.0.post0\n + pytz==2024.2\n + scipy==1.13.1\n + setuptools==75.6.0\n + six==1.17.0\n + tomli==2.2.1\n + typing-extensions==4.12.2\n + tzdata==2024.2\n + zipp==3.21.0\nResolved 63 packages in 3.81s\nPrepared 1 package in 5.05s\nInstalled 57 packages in 547ms\n + annexremote==1.6.6\n + appdirs==1.4.4\n + backports-tarfile==1.2.0\n + beautifulsoup4==4.12.3\n + boto==2.49.0\n + certifi==2024.12.14\n + cffi==1.17.1\n + chardet==5.2.0\n + charset-normalizer==3.4.1\n + citeproc-py==0.6.0\n + cryptography==44.0.0\n + datalad==0.14.0 (from file:///home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b3c3916449e13b8e3de4ecf4e0aee59b7fbc28f8)\n + deprecated==1.2.15\n + distro==1.9.0\n + duecredit==0.10.2\n + exifread==3.0.0\n + fasteners==0.19\n + httpretty==1.1.4\n + humanize==4.11.0\n + idna==3.10\n + importlib-metadata==8.5.0\n + iso8601==2.1.0\n + jaraco-classes==3.4.0\n + jaraco-context==6.0.1\n + jaraco-functools==4.1.0\n + jeepney==0.8.0\n + jsmin==3.0.1\n + keyring==25.6.0\n + keyrings-alt==5.0.2\n + looseversion==1.3.0\n + lxml==5.3.0\n + more-itertools==10.5.0\n + msgpack==1.1.0\n + multidict==6.1.0\n + mutagen==1.47.0\n + nose==1.3.7\n + patool==1.15.0\n + pillow==11.0.0\n + propcache==0.2.1\n + pycparser==2.22\n + pygithub==2.5.0\n + pyjwt==2.10.1\n + pynacl==1.5.0\n + pyperclip==1.9.0\n + python-xmp-toolkit==2.0.2\n + pyyaml==6.0.2\n + requests==2.32.3\n + requests-ftp==0.3.1\n + secretstorage==3.3.3\n + simplejson==3.19.3\n + soupsieve==2.6\n + tqdm==4.67.1\n + urllib3==1.26.20\n + vcrpy==7.0.0\n + whoosh==2.7.4\n + wrapt==1.17.0\n + yarl==1.18.3\nResolved 97 packages in 378ms\nInstalled 32 packages in 1.26s\n + alabaster==0.7.16\n + asv==0.6.4\n + asv-runner==0.2.1\n + babel==2.16.0\n + build==1.2.2.post1\n + distlib==0.3.9\n + docutils==0.21.2\n + filelock==3.16.1\n + gprof2dot==2024.6.6\n + imagesize==1.4.1\n + jinja2==3.1.5\n + json5==0.10.0\n + markupsafe==3.0.2\n + nose-timer==1.0.1\n + platformdirs==4.3.6\n + psutil==6.1.1\n + pygments==2.18.0\n + pympler==1.1\n + pypandoc==1.14\n + pyproject-hooks==1.2.0\n + snowballstemmer==2.2.0\n + sphinx==7.4.7\n + sphinx-rtd-theme==3.0.2\n + sphinxcontrib-applehelp==2.0.0\n + sphinxcontrib-devhelp==2.0.0\n + sphinxcontrib-htmlhelp==2.1.0\n + sphinxcontrib-jquery==4.1\n + sphinxcontrib-jsmath==1.0.1\n + sphinxcontrib-qthelp==2.0.0\n + sphinxcontrib-serializinghtml==2.0.0\n + tabulate==0.9.0\n + virtualenv==20.28.0\n",
    "new_commit_res_code": 1,
    "new_commit_res_stdout": "============================= test session starts ==============================\nplatform linux -- Python 3.9.21, pytest-8.3.4, pluggy-1.5.0\nrootdir: /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b3c3916449e13b8e3de4ecf4e0aee59b7fbc28f8\nconfigfile: pyproject.toml\nplugins: cov-6.0.0\ncollected 23 items\n\nr2e_tests/test_1.py FFEFFFFF                                             [ 34%]\nr2e_tests/test_2.py EFF....EFFFFFFF                                      [100%]\n\n==================================== ERRORS ====================================\n__________________ ERROR at setup of test_procedure_discovery __________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b3c3916449e13b8e3de4ecf4e0aee59b7fbc28f8/r2e_tests/test_1.py, line 82\n  @skip_if(cond=on_windows and dl_cfg.obtain(\"datalad.repo.version\") < 6)\n  @with_tree(tree={\n      'code': {'datalad_test_proc.py': \"\"\"\\\n  import sys\n  import os.path as op\n  from datalad.api import save, Dataset\n\n  with open(op.join(sys.argv[1], 'fromproc.txt'), 'w') as f:\n      f.write('hello\\\\n')\n  save(dataset=Dataset(sys.argv[1]), path='fromproc.txt')\n  \"\"\"},\n      'cfg_yoda.sh': \"\"\"\\\n  #!/bin/bash\n  echo \"I am never to be ran but could be discovered\"\n  exit 1\n  \"\"\"})\n  @with_tempfile\n  def test_procedure_discovery(path, super_path):\nE       fixture 'super_path' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b3c3916449e13b8e3de4ecf4e0aee59b7fbc28f8/r2e_tests/test_1.py:82\n_______________________ ERROR at setup of test_something _______________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b3c3916449e13b8e3de4ecf4e0aee59b7fbc28f8/r2e_tests/test_2.py, line 72\n  @with_tree(tree=_dataset_config_template)\n  @with_tempfile(mkdir=True)\n  def test_something(path, new_home):\nE       fixture 'new_home' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b3c3916449e13b8e3de4ecf4e0aee59b7fbc28f8/r2e_tests/test_2.py:72\n_______________________ ERROR at setup of test_no_leaks ________________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b3c3916449e13b8e3de4ecf4e0aee59b7fbc28f8/r2e_tests/test_2.py, line 483\n  @with_tempfile()\n  @with_tempfile()\n  def test_no_leaks(path1, path2):\nE       fixture 'path1' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b3c3916449e13b8e3de4ecf4e0aee59b7fbc28f8/r2e_tests/test_2.py:483\n=================================== FAILURES ===================================\n______________________________ test_invalid_call _______________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annex02m4dq54'}\nfilename = '/tmp/datalad_temp_test_invalid_callm560uq3v'\n\n    @wraps(t)\n    def  _wrap_with_tempfile(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_invalid_call() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:757: TypeError\n__________________________________ test_dirty __________________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annexkla4wmdi'}\ntkwargs_ = {'dir': None, 'prefix': 'datalad_temp_tree_test_dirty'}\nd = '/tmp/datalad_temp_tree_test_dirtythi5sd4b'\n\n    @wraps(t)\n    def  _wrap_with_tree(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        tkwargs_ = get_tempfile_kwargs(tkwargs, prefix=\"tree\", wrapped=t)\n        d = tempfile.mkdtemp(**tkwargs_)\n        create_tree(d, tree, archives_leading_dir=archives_leading_dir)\n        try:\n>           return t(*(arg + (d,)), **kw)\nE           TypeError: test_dirty() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:578: TypeError\n_________________________________ test_configs _________________________________\n\nargs = (), kwargs = {'path': '/tmp/tmp-page2annex28vnh4jy'}\n\n    @wraps(func)\n    def  _wrap_skip_if(*args, **kwargs):\n        if cond:\n            if method == 'raise':\n                raise SkipTest(msg if msg else \"condition was True\")\n            elif method == 'pass':\n                print(msg if msg else \"condition was True\")\n                return\n>       return func(*args, **kwargs)\n\ndatalad/tests/utils.py:259: \n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \n\narg = (), kw = {'path': '/tmp/tmp-page2annex28vnh4jy'}\ntkwargs_ = {'dir': None, 'prefix': 'datalad_temp_tree_test_configs'}\nd = '/tmp/datalad_temp_tree_test_configsb14ur0rv'\n\n    @wraps(t)\n    def  _wrap_with_tree(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        tkwargs_ = get_tempfile_kwargs(tkwargs, prefix=\"tree\", wrapped=t)\n        d = tempfile.mkdtemp(**tkwargs_)\n        create_tree(d, tree, archives_leading_dir=archives_leading_dir)\n        try:\n>           return t(*(arg + (d,)), **kw)\nE           TypeError: test_configs() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:578: TypeError\n_________________________________ test_spaces __________________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annex997oefg6'}\ntkwargs_ = {'dir': None, 'prefix': 'datalad_temp_tree_test_spaces'}\nd = '/tmp/datalad_temp_tree_test_spaces2vew206h'\n\n    @wraps(t)\n    def  _wrap_with_tree(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        tkwargs_ = get_tempfile_kwargs(tkwargs, prefix=\"tree\", wrapped=t)\n        d = tempfile.mkdtemp(**tkwargs_)\n        create_tree(d, tree, archives_leading_dir=archives_leading_dir)\n        try:\n>           return t(*(arg + (d,)), **kw)\nE           TypeError: test_spaces() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:578: TypeError\n_________________________________ test_quoting _________________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annexyftxj2xl'}\ntkwargs_ = {'dir': None, 'prefix': 'datalad_temp_tree_test_quoting'}\nd = '/tmp/datalad_temp_tree_test_quotingcft7veaz'\n\n    @wraps(t)\n    def  _wrap_with_tree(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        tkwargs_ = get_tempfile_kwargs(tkwargs, prefix=\"tree\", wrapped=t)\n        d = tempfile.mkdtemp(**tkwargs_)\n        create_tree(d, tree, archives_leading_dir=archives_leading_dir)\n        try:\n>           return t(*(arg + (d,)), **kw)\nE           TypeError: test_quoting() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:578: TypeError\n________________________________ test_text2git _________________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annexb9uv5imp'}\ntkwargs_ = {'dir': None, 'prefix': 'datalad_temp_tree_test_text2git'}\nd = '/tmp/datalad_temp_tree_test_text2gitqs_4xj1f'\n\n    @wraps(t)\n    def  _wrap_with_tree(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        tkwargs_ = get_tempfile_kwargs(tkwargs, prefix=\"tree\", wrapped=t)\n        d = tempfile.mkdtemp(**tkwargs_)\n        create_tree(d, tree, archives_leading_dir=archives_leading_dir)\n        try:\n>           return t(*(arg + (d,)), **kw)\nE           TypeError: test_text2git() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:578: TypeError\n__________________________ test_name_with_underscore ___________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annex42xm0q5t'}\ntkwargs_ = {'dir': None, 'prefix': 'datalad_temp_tree_test_name_with_underscore'}\nd = '/tmp/datalad_temp_tree_test_name_with_underscoreo0p70fzj'\n\n    @wraps(t)\n    def  _wrap_with_tree(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        tkwargs_ = get_tempfile_kwargs(tkwargs, prefix=\"tree\", wrapped=t)\n        d = tempfile.mkdtemp(**tkwargs_)\n        create_tree(d, tree, archives_leading_dir=archives_leading_dir)\n        try:\n>           return t(*(arg + (d,)), **kw)\nE           TypeError: test_name_with_underscore() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:578: TypeError\n________________________________ test_crazy_cfg ________________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annex_m6nv9jw'}\ntkwargs_ = {'dir': None, 'prefix': 'datalad_temp_tree_test_crazy_cfg'}\nd = '/tmp/datalad_temp_tree_test_crazy_cfgvqizrqf5'\n\n    @wraps(t)\n    def  _wrap_with_tree(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        tkwargs_ = get_tempfile_kwargs(tkwargs, prefix=\"tree\", wrapped=t)\n        d = tempfile.mkdtemp(**tkwargs_)\n        create_tree(d, tree, archives_leading_dir=archives_leading_dir)\n        try:\n>           return t(*(arg + (d,)), **kw)\nE           TypeError: test_crazy_cfg() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:578: TypeError\n_________________________________ test_obtain __________________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annexw86lapxc'}\nfilename = '/tmp/datalad_temp_test_obtainx5sv3bao'\n\n    @wraps(t)\n    def  _wrap_with_tempfile(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_obtain() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:757: TypeError\n______________________ test_no_local_write_if_no_dataset _______________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annexdjjy4asn'}\nfilename = '/tmp/datalad_temp_test_no_local_write_if_no_datasetfyv20w3q'\n\n    @wraps(t)\n    def  _wrap_with_tempfile(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_no_local_write_if_no_dataset() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:757: TypeError\n___________________________ test_dataset_local_mode ____________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annexfyk3p1ok'}\nfilename = '/tmp/datalad_temp_test_dataset_local_mode7umlc1zn'\n\n    @wraps(t)\n    def  _wrap_with_tempfile(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_dataset_local_mode() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:757: TypeError\n________________________ test_dataset_systemglobal_mode ________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annexhtuiu2y8'}\nfilename = '/tmp/datalad_temp_test_dataset_systemglobal_modexmhx3m88'\n\n    @wraps(t)\n    def  _wrap_with_tempfile(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_dataset_systemglobal_mode() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:757: TypeError\n______________________________ test_global_config ______________________________\n\n    def test_global_config():\n    \n        # from within tests, global config should be read from faked $HOME (see\n        # setup_package)\n        glb_cfg_file = Path(os.path.expanduser('~')) / '.gitconfig'\n>       assert any(glb_cfg_file.samefile(Path(p)) for p in dl_cfg._stores['git']['files'])\nE       assert False\nE        +  where False = any(<generator object test_global_config.<locals>.<genexpr> at 0x7042e3975430>)\n\nr2e_tests/test_2.py:563: AssertionError\n__________________________________ test_bare ___________________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annexw329_ema'}\nfilename = '/tmp/datalad_temp_test_bareg433wm_d'\n\n    @wraps(t)\n    def  _wrap_with_tempfile(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_bare() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:757: TypeError\n__________________________ test_write_config_section ___________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annex5iuj0tmc'}\nfilename = '/tmp/datalad_temp_test_write_config_section86a8of1w'\n\n    @wraps(t)\n    def  _wrap_with_tempfile(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_write_config_section() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:757: TypeError\n__________________________ test_external_modification __________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annexwl748tmn'}\nfilename = '/tmp/datalad_temp_test_external_modificationge34r6qn'\n\n    @wraps(t)\n    def  _wrap_with_tempfile(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_external_modification() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:757: TypeError\n==================================== PASSES ====================================\n___________________________ test_from_env_overrides ____________________________\n------------------------------ Captured log call -------------------------------\nWARNING  datalad.config:config.py:132 Failed to load DATALAD_CONFIG_OVERRIDES_JSON: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)\n_______________________________ test_rewrite_url _______________________________\n------------------------------ Captured log call -------------------------------\nWARNING  datalad.config:config.py:928 Ignoring URL rewrite configuration for 'conflict', multiple conflicting definitions exists: ['url.http://host1.insteadof', 'url.http://host2.insteadof']\nWARNING  datalad.config:config.py:928 Ignoring URL rewrite configuration for 'conflict2', multiple conflicting definitions exists: ['url.http://host3.insteadof', 'url.http://host4.insteadof']\n=========================== short test summary info ============================\nPASSED r2e_tests/test_2.py::test_from_env\nPASSED r2e_tests/test_2.py::test_from_env_overrides\nPASSED r2e_tests/test_2.py::test_overrides\nPASSED r2e_tests/test_2.py::test_rewrite_url\nERROR r2e_tests/test_1.py::test_procedure_discovery\nERROR r2e_tests/test_2.py::test_something\nERROR r2e_tests/test_2.py::test_no_leaks\nFAILED r2e_tests/test_1.py::test_invalid_call - TypeError: test_invalid_call(...\nFAILED r2e_tests/test_1.py::test_dirty - TypeError: test_dirty() got multiple...\nFAILED r2e_tests/test_1.py::test_configs - TypeError: test_configs() got mult...\nFAILED r2e_tests/test_1.py::test_spaces - TypeError: test_spaces() got multip...\nFAILED r2e_tests/test_1.py::test_quoting - TypeError: test_quoting() got mult...\nFAILED r2e_tests/test_1.py::test_text2git - TypeError: test_text2git() got mu...\nFAILED r2e_tests/test_1.py::test_name_with_underscore - TypeError: test_name_...\nFAILED r2e_tests/test_2.py::test_crazy_cfg - TypeError: test_crazy_cfg() got ...\nFAILED r2e_tests/test_2.py::test_obtain - TypeError: test_obtain() got multip...\nFAILED r2e_tests/test_2.py::test_no_local_write_if_no_dataset - TypeError: te...\nFAILED r2e_tests/test_2.py::test_dataset_local_mode - TypeError: test_dataset...\nFAILED r2e_tests/test_2.py::test_dataset_systemglobal_mode - TypeError: test_...\nFAILED r2e_tests/test_2.py::test_global_config - assert False\nFAILED r2e_tests/test_2.py::test_bare - TypeError: test_bare() got multiple v...\nFAILED r2e_tests/test_2.py::test_write_config_section - TypeError: test_write...\nFAILED r2e_tests/test_2.py::test_external_modification - TypeError: test_exte...\n==================== 16 failed, 4 passed, 3 errors in 3.02s ====================\n",
    "new_commit_res_stderr": "",
    "old_commit_res_code": 1,
    "old_commit_res_stdout": "============================= test session starts ==============================\nplatform linux -- Python 3.9.21, pytest-8.3.4, pluggy-1.5.0\nrootdir: /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b3c3916449e13b8e3de4ecf4e0aee59b7fbc28f8\nconfigfile: pyproject.toml\nplugins: cov-6.0.0\ncollected 23 items\n\nr2e_tests/test_1.py FFEFFFFF                                             [ 34%]\nr2e_tests/test_2.py EFF.F..EFFFFFFF                                      [100%]\n\n==================================== ERRORS ====================================\n__________________ ERROR at setup of test_procedure_discovery __________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b3c3916449e13b8e3de4ecf4e0aee59b7fbc28f8/r2e_tests/test_1.py, line 82\n  @skip_if(cond=on_windows and dl_cfg.obtain(\"datalad.repo.version\") < 6)\n  @with_tree(tree={\n      'code': {'datalad_test_proc.py': \"\"\"\\\n  import sys\n  import os.path as op\n  from datalad.api import save, Dataset\n\n  with open(op.join(sys.argv[1], 'fromproc.txt'), 'w') as f:\n      f.write('hello\\\\n')\n  save(dataset=Dataset(sys.argv[1]), path='fromproc.txt')\n  \"\"\"},\n      'cfg_yoda.sh': \"\"\"\\\n  #!/bin/bash\n  echo \"I am never to be ran but could be discovered\"\n  exit 1\n  \"\"\"})\n  @with_tempfile\n  def test_procedure_discovery(path, super_path):\nE       fixture 'super_path' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b3c3916449e13b8e3de4ecf4e0aee59b7fbc28f8/r2e_tests/test_1.py:82\n_______________________ ERROR at setup of test_something _______________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b3c3916449e13b8e3de4ecf4e0aee59b7fbc28f8/r2e_tests/test_2.py, line 72\n  @with_tree(tree=_dataset_config_template)\n  @with_tempfile(mkdir=True)\n  def test_something(path, new_home):\nE       fixture 'new_home' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b3c3916449e13b8e3de4ecf4e0aee59b7fbc28f8/r2e_tests/test_2.py:72\n_______________________ ERROR at setup of test_no_leaks ________________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b3c3916449e13b8e3de4ecf4e0aee59b7fbc28f8/r2e_tests/test_2.py, line 483\n  @with_tempfile()\n  @with_tempfile()\n  def test_no_leaks(path1, path2):\nE       fixture 'path1' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_b3c3916449e13b8e3de4ecf4e0aee59b7fbc28f8/r2e_tests/test_2.py:483\n=================================== FAILURES ===================================\n______________________________ test_invalid_call _______________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annexcyh6u0hg'}\nfilename = '/tmp/datalad_temp_test_invalid_call21j64d6r'\n\n    @wraps(t)\n    def  _wrap_with_tempfile(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_invalid_call() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:757: TypeError\n__________________________________ test_dirty __________________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annexohy4j7n5'}\ntkwargs_ = {'dir': None, 'prefix': 'datalad_temp_tree_test_dirty'}\nd = '/tmp/datalad_temp_tree_test_dirty6t4hv7n3'\n\n    @wraps(t)\n    def  _wrap_with_tree(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        tkwargs_ = get_tempfile_kwargs(tkwargs, prefix=\"tree\", wrapped=t)\n        d = tempfile.mkdtemp(**tkwargs_)\n        create_tree(d, tree, archives_leading_dir=archives_leading_dir)\n        try:\n>           return t(*(arg + (d,)), **kw)\nE           TypeError: test_dirty() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:578: TypeError\n_________________________________ test_configs _________________________________\n\nargs = (), kwargs = {'path': '/tmp/tmp-page2annexgdh43qj0'}\n\n    @wraps(func)\n    def  _wrap_skip_if(*args, **kwargs):\n        if cond:\n            if method == 'raise':\n                raise SkipTest(msg if msg else \"condition was True\")\n            elif method == 'pass':\n                print(msg if msg else \"condition was True\")\n                return\n>       return func(*args, **kwargs)\n\ndatalad/tests/utils.py:259: \n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \n\narg = (), kw = {'path': '/tmp/tmp-page2annexgdh43qj0'}\ntkwargs_ = {'dir': None, 'prefix': 'datalad_temp_tree_test_configs'}\nd = '/tmp/datalad_temp_tree_test_configsb23v3zqm'\n\n    @wraps(t)\n    def  _wrap_with_tree(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        tkwargs_ = get_tempfile_kwargs(tkwargs, prefix=\"tree\", wrapped=t)\n        d = tempfile.mkdtemp(**tkwargs_)\n        create_tree(d, tree, archives_leading_dir=archives_leading_dir)\n        try:\n>           return t(*(arg + (d,)), **kw)\nE           TypeError: test_configs() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:578: TypeError\n_________________________________ test_spaces __________________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annexckeajw89'}\ntkwargs_ = {'dir': None, 'prefix': 'datalad_temp_tree_test_spaces'}\nd = '/tmp/datalad_temp_tree_test_spacesdv5pec8i'\n\n    @wraps(t)\n    def  _wrap_with_tree(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        tkwargs_ = get_tempfile_kwargs(tkwargs, prefix=\"tree\", wrapped=t)\n        d = tempfile.mkdtemp(**tkwargs_)\n        create_tree(d, tree, archives_leading_dir=archives_leading_dir)\n        try:\n>           return t(*(arg + (d,)), **kw)\nE           TypeError: test_spaces() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:578: TypeError\n_________________________________ test_quoting _________________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annexwmais7z0'}\ntkwargs_ = {'dir': None, 'prefix': 'datalad_temp_tree_test_quoting'}\nd = '/tmp/datalad_temp_tree_test_quoting4_16a9jn'\n\n    @wraps(t)\n    def  _wrap_with_tree(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        tkwargs_ = get_tempfile_kwargs(tkwargs, prefix=\"tree\", wrapped=t)\n        d = tempfile.mkdtemp(**tkwargs_)\n        create_tree(d, tree, archives_leading_dir=archives_leading_dir)\n        try:\n>           return t(*(arg + (d,)), **kw)\nE           TypeError: test_quoting() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:578: TypeError\n________________________________ test_text2git _________________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annex3tv4fpsr'}\ntkwargs_ = {'dir': None, 'prefix': 'datalad_temp_tree_test_text2git'}\nd = '/tmp/datalad_temp_tree_test_text2git9r9qok8t'\n\n    @wraps(t)\n    def  _wrap_with_tree(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        tkwargs_ = get_tempfile_kwargs(tkwargs, prefix=\"tree\", wrapped=t)\n        d = tempfile.mkdtemp(**tkwargs_)\n        create_tree(d, tree, archives_leading_dir=archives_leading_dir)\n        try:\n>           return t(*(arg + (d,)), **kw)\nE           TypeError: test_text2git() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:578: TypeError\n__________________________ test_name_with_underscore ___________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annexy2pdch4f'}\ntkwargs_ = {'dir': None, 'prefix': 'datalad_temp_tree_test_name_with_underscore'}\nd = '/tmp/datalad_temp_tree_test_name_with_underscoremct71wou'\n\n    @wraps(t)\n    def  _wrap_with_tree(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        tkwargs_ = get_tempfile_kwargs(tkwargs, prefix=\"tree\", wrapped=t)\n        d = tempfile.mkdtemp(**tkwargs_)\n        create_tree(d, tree, archives_leading_dir=archives_leading_dir)\n        try:\n>           return t(*(arg + (d,)), **kw)\nE           TypeError: test_name_with_underscore() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:578: TypeError\n________________________________ test_crazy_cfg ________________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annexa7lyfipo'}\ntkwargs_ = {'dir': None, 'prefix': 'datalad_temp_tree_test_crazy_cfg'}\nd = '/tmp/datalad_temp_tree_test_crazy_cfgqjjazhg0'\n\n    @wraps(t)\n    def  _wrap_with_tree(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        tkwargs_ = get_tempfile_kwargs(tkwargs, prefix=\"tree\", wrapped=t)\n        d = tempfile.mkdtemp(**tkwargs_)\n        create_tree(d, tree, archives_leading_dir=archives_leading_dir)\n        try:\n>           return t(*(arg + (d,)), **kw)\nE           TypeError: test_crazy_cfg() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:578: TypeError\n_________________________________ test_obtain __________________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annexmzgm8mgu'}\nfilename = '/tmp/datalad_temp_test_obtainc18e9f6x'\n\n    @wraps(t)\n    def  _wrap_with_tempfile(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_obtain() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:757: TypeError\n___________________________ test_from_env_overrides ____________________________\n\n    def test_from_env_overrides():\n        cfg = ConfigManager()\n        assert_not_in(\"datalad.FoO\", cfg)\n    \n        # Some details, like case and underscores, cannot be handled by the direct\n        # environment variable mapping.\n        with patch.dict(\"os.environ\",\n                        {\"DATALAD_FOO\": \"val\"}):\n            cfg.reload()\n            assert_not_in(\"datalad.FoO\", cfg)\n            assert_equal(cfg[\"datalad.foo\"], \"val\")\n    \n        # But they can be handled via DATALAD_CONFIG_OVERRIDES_JSON.\n        with patch.dict(\"os.environ\",\n                        {\"DATALAD_CONFIG_OVERRIDES_JSON\": '{\"datalad.FoO\": \"val\"}'}):\n            cfg.reload()\n>           assert_equal(cfg[\"datalad.FoO\"], \"val\")\n\nr2e_tests/test_2.py:380: \n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \n\nself = ConfigManager(['/tmp/tmp-page2annexkhd4gq8d/.gitconfig'])\nkey = 'datalad.FoO'\n\n    def __getitem__(self, key):\n>       return self._merged_store.__getitem__(key)\nE       KeyError: 'datalad.FoO'\n\ndatalad/config.py:546: KeyError\n______________________ test_no_local_write_if_no_dataset _______________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annex13nn0h05'}\nfilename = '/tmp/datalad_temp_test_no_local_write_if_no_datasetn9tm4fgc'\n\n    @wraps(t)\n    def  _wrap_with_tempfile(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_no_local_write_if_no_dataset() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:757: TypeError\n___________________________ test_dataset_local_mode ____________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annexavncl4rb'}\nfilename = '/tmp/datalad_temp_test_dataset_local_modeqb9bugak'\n\n    @wraps(t)\n    def  _wrap_with_tempfile(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_dataset_local_mode() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:757: TypeError\n________________________ test_dataset_systemglobal_mode ________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annexya6sjpm5'}\nfilename = '/tmp/datalad_temp_test_dataset_systemglobal_modeug6di8o5'\n\n    @wraps(t)\n    def  _wrap_with_tempfile(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_dataset_systemglobal_mode() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:757: TypeError\n______________________________ test_global_config ______________________________\n\n    def test_global_config():\n    \n        # from within tests, global config should be read from faked $HOME (see\n        # setup_package)\n        glb_cfg_file = Path(os.path.expanduser('~')) / '.gitconfig'\n>       assert any(glb_cfg_file.samefile(Path(p)) for p in dl_cfg._stores['git']['files'])\nE       assert False\nE        +  where False = any(<generator object test_global_config.<locals>.<genexpr> at 0x7c8e23439d60>)\n\nr2e_tests/test_2.py:563: AssertionError\n__________________________________ test_bare ___________________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annexbd5qvi0y'}\nfilename = '/tmp/datalad_temp_test_bared72a71v2'\n\n    @wraps(t)\n    def  _wrap_with_tempfile(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_bare() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:757: TypeError\n__________________________ test_write_config_section ___________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annexp1tp2u6u'}\nfilename = '/tmp/datalad_temp_test_write_config_sectionsfkykk_7'\n\n    @wraps(t)\n    def  _wrap_with_tempfile(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_write_config_section() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:757: TypeError\n__________________________ test_external_modification __________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annexxe6o0eqh'}\nfilename = '/tmp/datalad_temp_test_external_modification0upveld3'\n\n    @wraps(t)\n    def  _wrap_with_tempfile(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_external_modification() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:757: TypeError\n==================================== PASSES ====================================\n_______________________________ test_rewrite_url _______________________________\n------------------------------ Captured log call -------------------------------\nWARNING  datalad.config:config.py:928 Ignoring URL rewrite configuration for 'conflict', multiple conflicting definitions exists: ['url.http://host1.insteadof', 'url.http://host2.insteadof']\nWARNING  datalad.config:config.py:928 Ignoring URL rewrite configuration for 'conflict2', multiple conflicting definitions exists: ['url.http://host3.insteadof', 'url.http://host4.insteadof']\n=========================== short test summary info ============================\nPASSED r2e_tests/test_2.py::test_from_env\nPASSED r2e_tests/test_2.py::test_overrides\nPASSED r2e_tests/test_2.py::test_rewrite_url\nERROR r2e_tests/test_1.py::test_procedure_discovery\nERROR r2e_tests/test_2.py::test_something\nERROR r2e_tests/test_2.py::test_no_leaks\nFAILED r2e_tests/test_1.py::test_invalid_call - TypeError: test_invalid_call(...\nFAILED r2e_tests/test_1.py::test_dirty - TypeError: test_dirty() got multiple...\nFAILED r2e_tests/test_1.py::test_configs - TypeError: test_configs() got mult...\nFAILED r2e_tests/test_1.py::test_spaces - TypeError: test_spaces() got multip...\nFAILED r2e_tests/test_1.py::test_quoting - TypeError: test_quoting() got mult...\nFAILED r2e_tests/test_1.py::test_text2git - TypeError: test_text2git() got mu...\nFAILED r2e_tests/test_1.py::test_name_with_underscore - TypeError: test_name_...\nFAILED r2e_tests/test_2.py::test_crazy_cfg - TypeError: test_crazy_cfg() got ...\nFAILED r2e_tests/test_2.py::test_obtain - TypeError: test_obtain() got multip...\nFAILED r2e_tests/test_2.py::test_from_env_overrides - KeyError: 'datalad.FoO'\nFAILED r2e_tests/test_2.py::test_no_local_write_if_no_dataset - TypeError: te...\nFAILED r2e_tests/test_2.py::test_dataset_local_mode - TypeError: test_dataset...\nFAILED r2e_tests/test_2.py::test_dataset_systemglobal_mode - TypeError: test_...\nFAILED r2e_tests/test_2.py::test_global_config - assert False\nFAILED r2e_tests/test_2.py::test_bare - TypeError: test_bare() got multiple v...\nFAILED r2e_tests/test_2.py::test_write_config_section - TypeError: test_write...\nFAILED r2e_tests/test_2.py::test_external_modification - TypeError: test_exte...\n==================== 17 failed, 3 passed, 3 errors in 1.43s ====================\n",
    "old_commit_res_stderr": ""
}
datalad/config.py datalad/interface/tests/test_run_procedure.py datalad/tests/test_config.py docs/source/config.rst
We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
[ISSUE]
**Title:** Environment Variable `DATALAD_CONFIG_OVERRIDES` Not Applied, Causing Configuration Key Error

**Description:**
When setting the `DATALAD_CONFIG_OVERRIDES` environment variable with JSON-formatted overrides, the configuration does not apply these overrides correctly. As a result, accessing certain configuration keys raises a `KeyError`, preventing expected configurations from being recognized.

**Example Code:**
```python
import os
from datalad.config import ConfigManager

# Set the environment variable with JSON overrides
os.environ['DATALAD_CONFIG_OVERRIDES'] = '{"datalad.FoO": "val"}'

# Initialize and reload the configuration
cfg = ConfigManager()
cfg.reload()

# Attempt to access the overridden configuration key
print(cfg["datalad.FoO"])  # This raises KeyError
```

**Expected Behavior:**
The configuration key `datalad.FoO` should be set to `"val"` as specified in the `DATALAD_CONFIG_OVERRIDES` environment variable, allowing the program to access it without errors.

**Actual Behavior:**
Accessing `cfg["datalad.FoO"]` raises a `KeyError` because the overrides specified in `DATALAD_CONFIG_OVERRIDES` are not applied correctly.

[/ISSUE]
--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.

--- BEGIN FILE ---
```
### datalad/config.py
# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-
# ex: set sts=4 ts=4 sw=4 noet:
# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
#
#   See COPYING file distributed along with the datalad package for the
#   copyright and license terms.
#
# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
"""
"""

from collections import namedtuple

import json
import threading
from fasteners import InterProcessLock
from functools import lru_cache
import datalad
from datalad.consts import (
    DATASET_CONFIG_FILE,
)
from datalad.cmd import (
    GitWitlessRunner,
    StdOutErrCapture,
)
from datalad.dochelpers import exc_str

import re
import os
from pathlib import Path

import logging
lgr = logging.getLogger('datalad.config')

cfg_kv_regex = re.compile(r'(^.*)\n(.*)$', flags=re.MULTILINE)
cfg_section_regex = re.compile(r'(.*)\.[^.]+')
cfg_sectionoption_regex = re.compile(r'(.*)\.([^.]+)')


_where_reload_doc = """
        where : {'dataset', 'local', 'global', 'override'}, optional
          Indicator which configuration file to modify. 'dataset' indicates the
          persistent configuration in .datalad/config of a dataset; 'local'
          the configuration of a dataset's Git repository in .git/config;
          'global' refers to the general configuration that is not specific to
          a single repository (usually in $USER/.gitconfig); 'override'
          limits the modification to the ConfigManager instance, and the
          assigned value overrides any setting from any other source.
        reload : bool
          Flag whether to reload the configuration from file(s) after
          modification. This can be disable to make multiple sequential
          modifications slightly more efficient.""".lstrip()

# Selection of os.stat_result fields we care to collect/compare to judge
# on either file has changed to warrant reload of configuration.
_stat_result = namedtuple('_stat_result', 'st_ino st_size st_ctime st_mtime')


# we cannot import external_versions here, as the cfg comes before anything
# and we would have circular imports
@lru_cache()
def get_git_version(runner=None):
    """Return version of available git"""
    runner = runner or GitWitlessRunner()
    return runner.run('git version'.split(),
                      protocol=StdOutErrCapture)['stdout'].split()[2]


def _where_reload(obj):
    """Helper decorator to simplify providing repetitive docstring"""
    obj.__doc__ = obj.__doc__ % _where_reload_doc
    return obj


# TODO document and make "public" (used in GitRepo too)
def _parse_gitconfig_dump(dump, cwd=None, multi_value=True):
    """Parse a dump-string from `git config -z --list`

    Parameters
    ----------
    dump : str
      Null-byte separated output
    cwd : path-like, optional
      Use this path to convert relative paths for origin reports
      into absolute paths
    multi_value : bool, optional
      If True, report values from multiple specifications of the
      same key as a tuple of values assigned to this key. Otherwise,
      the last configuration is reported.
    """
    dct = {}
    fileset = set()
    for line in dump.split('\0'):
        if not line:
            continue
        if line.startswith('file:'):
            # origin line
            fname = Path(line[5:])
            if not fname.is_absolute():
                fname = Path(cwd) / fname if cwd else Path.cwd() / fname
            fileset.add(fname)
            continue
        if line.startswith('command line:'):
            # nothing we could handle
            continue
        kv_match = cfg_kv_regex.match(line)
        if kv_match:
            k, v = kv_match.groups()
        else:
            # could be just a key without = value, which git treats as True
            # if asked for a bool
            k, v = line, None
        present_v = dct.get(k, None)
        if present_v is None or not multi_value:
            dct[k] = v
        else:
            if isinstance(present_v, tuple):
                dct[k] = present_v + (v,)
            else:
                dct[k] = (present_v, v)
    return dct, fileset


def _update_from_env(store):
    overrides = {}
    dct = {}
    for k in os.environ:
        if k == "DATALAD_CONFIG_OVERRIDES":
            try:
                overrides = json.loads(os.environ[k])
            except json.decoder.JSONDecodeError as exc:
                lgr.warning("Failed to load DATALAD_CONFIG_OVERRIDES: %s",
                            exc)
        elif k.startswith('DATALAD_'):
            dct[k.replace('__', '-').replace('_', '.').lower()] = os.environ[k]
    if overrides:
        store.update(overrides)
    store.update(dct)


def anything2bool(val):
    if hasattr(val, 'lower'):
        val = val.lower()
    if val in {"off", "no", "false", "0"} or not bool(val):
        return False
    elif val in {"on", "yes", "true", True} \
            or (hasattr(val, 'isdigit') and val.isdigit() and int(val)) \
            or isinstance(val, int) and val:
        return True
    else:
        raise TypeError(
            "Got value %s which could not be interpreted as a boolean"
            % repr(val))


class ConfigManager(object):
    """Thin wrapper around `git-config` with support for a dataset configuration.

    The general idea is to have an object that is primarily used to read/query
    configuration option.  Upon creation, current configuration is read via one
    (or max two, in the case of the presence of dataset-specific configuration)
    calls to `git config`.  If this class is initialized with a Dataset
    instance, it supports reading and writing configuration from
    ``.datalad/config`` inside a dataset too. This file is committed to Git and
    hence useful to ship certain configuration items with a dataset.

    The API aims to provide the most significant read-access API of a
    dictionary, the Python ConfigParser, and GitPython's config parser
    implementations.

    This class is presently not capable of efficiently writing multiple
    configurations items at once.  Instead, each modification results in a
    dedicated call to `git config`. This author thinks this is OK, as he
    cannot think of a situation where a large number of items need to be
    written during normal operation.

    Each instance carries a public `overrides` attribute. This dictionary
    contains variables that override any setting read from a file. The overrides
    are persistent across reloads.

    Any DATALAD_* environment variable is also presented as a configuration
    item. Settings read from environment variables are not stored in any of the
    configuration files, but are read dynamically from the environment at each
    `reload()` call. Their values take precedence over any specification in
    configuration files, and even overrides.

    Parameters
    ----------
    dataset : Dataset, optional
      If provided, all `git config` calls are executed in this dataset's
      directory. Moreover, any modifications are, by default, directed to
      this dataset's configuration file (which will be created on demand)
    overrides : dict, optional
      Variable overrides, see general class documentation for details.
    source : {'any', 'local', 'dataset', 'dataset-local'}, optional
      Which sources of configuration setting to consider. If 'dataset',
      configuration items are only read from a dataset's persistent
      configuration file, if any is present (the one in ``.datalad/config``, not
      ``.git/config``); if 'local', any non-committed source is considered
      (local and global configuration in Git config's terminology);
      if 'dataset-local', persistent dataset configuration and local, but
      not global or system configuration are considered; if 'any'
      all possible sources of configuration are considered.
    """

    _checked_git_identity = False

    # Lock for running changing operation across multiple threads.
    # Since config itself to the same path could
    # potentially be created independently in multiple threads, and we might be
    # modifying global config as well, making lock static should not allow more than
    # one thread to  write at a time, even if to different repositories.
    _run_lock = threading.Lock()

    def __init__(self, dataset=None, overrides=None, source='any'):
        if source not in ('any', 'local', 'dataset', 'dataset-local'):
            raise ValueError(
                'Unknown ConfigManager(source=) setting: {}'.format(source))
        store = dict(
            # store in a simple dict
            # no subclassing, because we want to be largely read-only, and implement
            # config writing separately
            cfg={},
            # track the files that jointly make up the config in this store
            files=set(),
            # and their modification times to be able to avoid needless unforced reloads
            stats=None,
        )
        self._stores = dict(
            # populated with info from git
            git=store,
            # only populated with info from commited dataset config
            dataset=store.copy(),
        )
        # merged representation (the only one that existed pre datalad 0.14)
        # will be built on initial reload
        self._merged_store = {}

        self._repo_dot_git = None
        self._repo_pathobj = None
        if dataset:
            if hasattr(dataset, 'dot_git'):
                self._repo_dot_git = dataset.dot_git
                self._repo_pathobj = dataset.pathobj
            elif dataset.repo:
                self._repo_dot_git = dataset.repo.dot_git
                self._repo_pathobj = dataset.repo.pathobj

        self._config_cmd = ['git', 'config']
        # public dict to store variables that always override any setting
        # read from a file
        # `hasattr()` is needed because `datalad.cfg` is generated upon first module
        # import, hence when this code runs first, there cannot be any config manager
        # to inherit from
        self.overrides = datalad.cfg.overrides.copy() if hasattr(datalad, 'cfg') else {}
        if overrides is not None:
            self.overrides.update(overrides)
        if dataset is None:
            if source in ('dataset', 'dataset-local'):
                raise ValueError(
                    'ConfigManager configured to read dataset only, '
                    'but no dataset given')
            # The caller didn't specify a repository. Unset the git directory
            # when calling 'git config' to prevent a repository in the current
            # working directory from leaking configuration into the output.
            self._config_cmd = ['git', '--git-dir=', 'config']

        self._src_mode = source
        run_kwargs = dict()
        self._runner = None
        if dataset is not None:
            if hasattr(dataset, '_git_runner'):
                self._runner = dataset._git_runner
            elif dataset.repo:
                self._runner = dataset.repo._git_runner
            else:
                # make sure we run the git config calls in the dataset
                # to pick up the right config files
                run_kwargs['cwd'] = dataset.path
        if self._runner is None:
            self._runner = GitWitlessRunner(**run_kwargs)

        self.reload(force=True)

        if not ConfigManager._checked_git_identity:
            for cfg, envs in (
                    ('user.name', ('GIT_AUTHOR_NAME', 'GIT_COMMITTER_NAME')),
                    ('user.email', ('GIT_AUTHOR_EMAIL', 'GIT_COMMITTER_EMAIL'))):
                if cfg not in self \
                        and not any(e in os.environ for e in envs):
                    lgr.warning(
                        "It is highly recommended to configure Git before using "
                        "DataLad. Set both 'user.name' and 'user.email' "
                        "configuration variables."
                    )
            ConfigManager._checked_git_identity = True

    def reload(self, force=False):
        """Reload all configuration items from the configured sources

        If `force` is False, all files configuration was previously read from
        are checked for differences in the modification times. If no difference
        is found for any file no reload is performed. This mechanism will not
        detect newly created global configuration files, use `force` in this case.
        """
        run_args = ['-z', '-l', '--show-origin']

        # update from desired config sources only
        # 2-step strategy:
        #   - load datalad dataset config from dataset
        #   - load git config from all supported by git sources
        # in doing so we always stay compatible with where Git gets its
        # config from, but also allow to override persistent information
        # from dataset locally or globally

        # figure out what needs to be reloaded at all
        to_run = {}
        # committed dataset config
        dataset_cfgfile = self._repo_pathobj / DATASET_CONFIG_FILE \
            if self._repo_pathobj else None
        if (self._src_mode != 'local' and
                dataset_cfgfile and
                dataset_cfgfile.exists()) and (
                force or self._need_reload(self._stores['dataset'])):
            to_run['dataset'] = run_args + ['--file', str(dataset_cfgfile)]

        if self._src_mode != 'dataset' and (
                force or self._need_reload(self._stores['git'])):
            to_run['git'] = run_args + ['--local'] \
                if self._src_mode == 'dataset-local' \
                else run_args

        # reload everything that was found todo
        while to_run:
            store_id, runargs = to_run.popitem()
            self._stores[store_id] = self._reload(runargs)

        # always update the merged representation, even if we did not reload
        # anything from a file. ENV or overrides could change independently
        # start with the commit dataset config
        merged = self._stores['dataset']['cfg'].copy()
        # local config always takes precedence
        merged.update(self._stores['git']['cfg'])
        # superimpose overrides
        merged.update(self.overrides)
        # override with environment variables, unless we only want to read the
        # dataset's commit config
        if self._src_mode != 'dataset':
            _update_from_env(merged)
        self._merged_store = merged

    def _need_reload(self, store):
        storestats = store['stats']
        if not storestats:
            return True

        # we have read files before
        # check if any file we read from has changed
        curstats = self._get_stats(store)
        return any(curstats[f] != storestats[f] for f in store['files'])

    def _reload(self, run_args):
        # query git-config
        stdout, stderr = self._run(
            run_args,
            protocol=StdOutErrCapture,
            # always expect git-config to output utf-8
            encoding='utf-8',
        )
        store = {}
        store['cfg'], store['files'] = _parse_gitconfig_dump(
            stdout, cwd=self._runner.cwd)

        # update stats of config files, they have just been discovered
        # and should still exist
        store['stats'] = self._get_stats(store)
        return store

    @staticmethod
    def _get_stats(store):
        stats = {}
        for f in store['files']:
            if f.exists:
                stat = f.stat()
                stats[f] = _stat_result(stat.st_ino, stat.st_size, stat.st_ctime, stat.st_mtime)
            else:
                stats[f] = None
        return stats

    @_where_reload
    def obtain(self, var, default=None, dialog_type=None, valtype=None,
               store=False, where=None, reload=True, **kwargs):
        """
        Convenience method to obtain settings interactively, if needed

        A UI will be used to ask for user input in interactive sessions.
        Questions to ask, and additional explanations can be passed directly
        as arguments, or retrieved from a list of pre-configured items.

        Additionally, this method allows for type conversion and storage
        of obtained settings. Both aspects can also be pre-configured.

        Parameters
        ----------
        var : str
          Variable name including any section like `git config` expects them,
          e.g. 'core.editor'
        default : any type
          In interactive sessions and if `store` is True, this default value
          will be presented to the user for confirmation (or modification).
          In all other cases, this value will be silently assigned unless
          there is an existing configuration setting.
        dialog_type : {'question', 'yesno', None}
          Which dialog type to use in interactive sessions. If `None`,
          pre-configured UI options are used.
        store : bool
          Whether to store the obtained value (or default)
        %s
        `**kwargs`
          Additional arguments for the UI function call, such as a question
          `text`.
        """
        # do local import, as this module is import prominently and the
        # could theroetically import all kind of weired things for type
        # conversion
        from datalad.interface.common_cfg import definitions as cfg_defs
        # fetch what we know about this variable
        cdef = cfg_defs.get(var, {})
        # type conversion setup
        if valtype is None and 'type' in cdef:
            valtype = cdef['type']
        if valtype is None:
            valtype = lambda x: x

        # any default?
        if default is None and 'default' in cdef:
            default = cdef['default']

        _value = None
        if var in self:
            # nothing needs to be obtained, it is all here already
            _value = self[var]
        elif store is False and default is not None:
            # nothing will be stored, and we have a default -> no user confirmation
            # we cannot use logging, because we want to use the config to confiugre
            # the logging
            #lgr.debug('using default {} for config setting {}'.format(default, var))
            _value = default

        if _value is not None:
            # we got everything we need and can exit early
            try:
                return valtype(_value)
            except Exception as e:
                raise ValueError(
                    "value '{}' of existing configuration for '{}' cannot be "
                    "converted to the desired type '{}' ({})".format(
                        _value, var, valtype, exc_str(e)))

        # now we need to try to obtain something from the user
        from datalad.ui import ui

        # configure UI
        dialog_opts = kwargs
        if dialog_type is None:  # no override
            # check for common knowledge on how to obtain a value
            if 'ui' in cdef:
                dialog_type = cdef['ui'][0]
                # pull standard dialog settings
                dialog_opts = cdef['ui'][1]
                # update with input
                dialog_opts.update(kwargs)

        if (not ui.is_interactive or dialog_type is None) and default is None:
            raise RuntimeError(
                "cannot obtain value for configuration item '{}', "
                "not preconfigured, no default, no UI available".format(var))

        if not hasattr(ui, dialog_type):
            raise ValueError("UI '{}' does not support dialog type '{}'".format(
                ui, dialog_type))

        # configure storage destination, if needed
        if store:
            if where is None and 'destination' in cdef:
                where = cdef['destination']
            if where is None:
                raise ValueError(
                    "request to store configuration item '{}', but no "
                    "storage destination specified".format(var))

        # obtain via UI
        dialog = getattr(ui, dialog_type)
        _value = dialog(default=default, **dialog_opts)

        if _value is None:
            # we got nothing
            if default is None:
                raise RuntimeError(
                    "could not obtain value for configuration item '{}', "
                    "not preconfigured, no default".format(var))
            # XXX maybe we should return default here, even it was returned
            # from the UI -- if that is even possible

        # execute type conversion before storing to check that we got
        # something that looks like what we want
        try:
            value = valtype(_value)
        except Exception as e:
            raise ValueError(
                "cannot convert user input `{}` to desired type ({})".format(
                    _value, exc_str(e)))
            # XXX we could consider "looping" until we have a value of proper
            # type in case of a user typo...

        if store:
            # store value as it was before any conversion, needs to be str
            # anyway
            # needs string conversion nevertheless, because default could come
            # in as something else
            self.add(var, '{}'.format(_value), where=where, reload=reload)
        return value

    def __repr__(self):
        # give full list of all tracked config files, plus overrides
        return "ConfigManager({}{})".format(
            [str(p) for p in self._stores['dataset']['files'].union(
                self._stores['git']['files'])],
            ', overrides={!r}'.format(self.overrides) if self.overrides else '',
        )

    def __str__(self):
        # give path of dataset, if there is any, plus overrides
        return "ConfigManager({}{})".format(
            self._repo_pathobj if self._repo_pathobj else '',
            'with overrides' if self.overrides else '',
        )

    #
    # Compatibility with dict API
    #
    def __len__(self):
        return len(self._merged_store)

    def __getitem__(self, key):
        return self._merged_store.__getitem__(key)

    def __contains__(self, key):
        return self._merged_store.__contains__(key)

    def keys(self):
        """Returns list of configuration item names"""
        return self._merged_store.keys()

    # XXX should this be *args?
    def get(self, key, default=None, get_all=False):
        """D.get(k[,d]) -> D[k] if k in D, else d.  d defaults to None.

        Parameters
        ----------
        default : optional
          Value to return when key is not present. `None` by default.
        get_all : bool, optional
          If True, return all values of multiple identical configuration keys.
          By default only the last specified value is returned.
        """
        try:
            val = self[key]
            if get_all or not isinstance(val, tuple):
                return val
            else:
                return val[-1]
        except KeyError:
            # return as-is, default could be a tuple, hence do not subject to
            # get_all processing
            return default

    def get_from_source(self, source, key, default=None):
        """Like get(), but a source can be specific.

        If `source` is 'dataset', only the commited configuration is queried,
        overrides are applied. In the case of 'local', the committed
        configuration is ignored, but overrides and configuration from
        environment variables are applied as usual.
        """
        if source not in ('dataset', 'local'):
            raise ValueError("source must be 'dataset' or 'local'")
        if source == 'dataset':
            return self.overrides.get(
                key,
                self._stores['dataset']['cfg'].get(
                    key,
                    default))
        else:
            if key not in self._stores['dataset']['cfg']:
                # the key is not in the committed config, hence we can
                # just report based on the merged representation
                return self.get(key, default)
            else:
                # expensive case, rebuild a config without the committed
                # dataset config contributing
                env = {}
                _update_from_env(env)
                return env.get(
                    key,
                    self.overrides.get(
                        key,
                        self._stores['local']['cfg'].get(
                            key,
                            default)))

    #
    # Compatibility with ConfigParser API
    #
    def sections(self):
        """Returns a list of the sections available"""
        return list(set([cfg_section_regex.match(k).group(1) for k in self._merged_store]))

    def options(self, section):
        """Returns a list of options available in the specified section."""
        opts = []
        for k in self._merged_store:
            sec, opt = cfg_sectionoption_regex.match(k).groups()
            if sec == section:
                opts.append(opt)
        return opts

    def has_section(self, section):
        """Indicates whether a section is present in the configuration"""
        for k in self._merged_store:
            if k.startswith(section):
                return True
        return False

    def has_option(self, section, option):
        """If the given section exists, and contains the given option"""
        for k in self._merged_store:
            sec, opt = cfg_sectionoption_regex.match(k).groups()
            if sec == section and opt == option:
                return True
        return False

    def _get_type(self, typefn, section, option):
        key = '.'.join([section, option])
        # Mimic the handling of get_value(..., default=None), while still going
        # through get() in order to get its default tuple handling.
        if key not in self:
            raise KeyError(key)
        return typefn(self.get(key))

    def getint(self, section, option):
        """A convenience method which coerces the option value to an integer"""
        return self._get_type(int, section, option)

    def getfloat(self, section, option):
        """A convenience method which coerces the option value to a float"""
        return self._get_type(float, section, option)

    def getbool(self, section, option, default=None):
        """A convenience method which coerces the option value to a bool

        Values "on", "yes", "true" and any int!=0 are considered True
        Values which evaluate to bool False, "off", "no", "false" are considered
        False
        TypeError is raised for other values.
        """
        key = '.'.join([section, option])
        # Mimic the handling of get_value(..., default=None), while still going
        # through get() in order to get its default tuple handling.
        if default is None and key not in self:
            raise KeyError(key)
        val = self.get(key, default=default)
        if val is None:  # no value at all, git treats it as True
            return True
        return anything2bool(val)

    # this is a hybrid of ConfigParser and dict API
    def items(self, section=None):
        """Return a list of (name, value) pairs for each option

        Optionally limited to a given section.
        """
        if section is None:
            return self._merged_store.items()
        return [(k, v) for k, v in self._merged_store.items()
                if cfg_section_regex.match(k).group(1) == section]

    #
    # Compatibility with GitPython's ConfigParser
    #
    def get_value(self, section, option, default=None):
        """Like `get()`, but with an optional default value

        If the default is not None, the given default value will be returned in
        case the option did not exist. This behavior imitates GitPython's
        config parser.
        """
        try:
            return self['.'.join((section, option))]
        except KeyError as e:
            # this strange dance is needed because gitpython does it this way
            if default is not None:
                return default
            else:
                raise e

    #
    # Modify configuration (proxy respective git-config call)
    #
    @_where_reload
    def _run(self, args, where=None, reload=False, **kwargs):
        """Centralized helper to run "git config" calls

        Parameters
        ----------
        args : list
          Arguments to pass for git config
        %s
        **kwargs
          Keywords arguments for Runner's call
        """
        if where:
            args = self._get_location_args(where) + args
        if '-l' in args:
            # we are just reading, no need to reload, no need to lock
            out = self._runner.run(self._config_cmd + args, **kwargs)
            return out['stdout'], out['stderr']

        # all other calls are modifications
        if '--file' in args:
            # all paths we are passing are absolute
            custom_file = Path(args[args.index('--file') + 1])
            custom_file.parent.mkdir(exist_ok=True)
        lockfile = None
        if self._repo_dot_git and ('--local' in args or '--file' in args):
            # modification of config in a dataset
            lockfile = self._repo_dot_git / 'config.dataladlock'
        else:
            # follow pattern in downloaders for lockfile location
            lockfile = Path(self.obtain('datalad.locations.cache')) \
                / 'locks' / 'gitconfig.lck'

        with ConfigManager._run_lock, InterProcessLock(lockfile, logger=lgr):
            out = self._runner.run(self._config_cmd + args, **kwargs)

        if reload:
            self.reload()
        return out['stdout'], out['stderr']

    def _get_location_args(self, where, args=None):
        if args is None:
            args = []
        cfg_labels = ('dataset', 'local', 'global', 'override')
        if where not in cfg_labels:
            raise ValueError(
                "unknown configuration label '{}' (not in {})".format(
                    where, cfg_labels))
        if where == 'dataset':
            if not self._repo_pathobj:
                raise ValueError(
                    'ConfigManager cannot store configuration to dataset, '
                    'none specified')
            dataset_cfgfile = self._repo_pathobj / DATASET_CONFIG_FILE
            args.extend(['--file', str(dataset_cfgfile)])
        elif where == 'global':
            args.append('--global')
        elif where == 'local':
            args.append('--local')
        return args

    @_where_reload
    def add(self, var, value, where='dataset', reload=True):
        """Add a configuration variable and value

        Parameters
        ----------
        var : str
          Variable name including any section like `git config` expects them, e.g.
          'core.editor'
        value : str
          Variable value
        %s"""
        if where == 'override':
            from datalad.utils import ensure_list
            val = ensure_list(self.overrides.pop(var, None))
            val.append(value)
            self.overrides[var] = val[0] if len(val) == 1 else val
            if reload:
                self.reload(force=True)
            return

        self._run(['--add', var, value], where=where, reload=reload,
                  protocol=StdOutErrCapture)

    @_where_reload
    def set(self, var, value, where='dataset', reload=True, force=False):
        """Set a variable to a value.

        In opposition to `add`, this replaces the value of `var` if there is
        one already.

        Parameters
        ----------
        var : str
          Variable name including any section like `git config` expects them, e.g.
          'core.editor'
        value : str
          Variable value
        force: bool
          if set, replaces all occurrences of `var` by a single one with the
          given `value`. Otherwise raise if multiple entries for `var` exist
          already
        %s"""
        if where == 'override':
            self.overrides[var] = value
            if reload:
                self.reload(force=True)
            return

        from datalad.support.gitrepo import to_options

        self._run(to_options(replace_all=force) + [var, value],
                  where=where, reload=reload, protocol=StdOutErrCapture)

    @_where_reload
    def rename_section(self, old, new, where='dataset', reload=True):
        """Rename a configuration section

        Parameters
        ----------
        old : str
          Name of the section to rename.
        new : str
          Name of the section to rename to.
        %s"""
        if where == 'override':
            self.overrides = {
                (new + k[len(old):]) if k.startswith(old + '.') else k: v
                for k, v in self.overrides.items()
            }
            if reload:
                self.reload(force=True)
            return

        self._run(['--rename-section', old, new], where=where, reload=reload)

    @_where_reload
    def remove_section(self, sec, where='dataset', reload=True):
        """Rename a configuration section

        Parameters
        ----------
        sec : str
          Name of the section to remove.
        %s"""
        if where == 'override':
            self.overrides = {
                k: v
                for k, v in self.overrides.items()
                if not k.startswith(sec + '.')
            }
            if reload:
                self.reload(force=True)
            return

        self._run(['--remove-section', sec], where=where, reload=reload)

    @_where_reload
    def unset(self, var, where='dataset', reload=True):
        """Remove all occurrences of a variable

        Parameters
        ----------
        var : str
          Name of the variable to remove
        %s"""
        if where == 'override':
            self.overrides.pop(var, None)
            if reload:
                self.reload(force=True)
            return

        # use unset all as it is simpler for now
        self._run(['--unset-all', var], where=where, reload=reload)


def rewrite_url(cfg, url):
    """Any matching 'url.<base>.insteadOf' configuration is applied

    Any URL that starts with such a configuration will be rewritten
    to start, instead, with <base>. When more than one insteadOf
    strings match a given URL, the longest match is used.

    Parameters
    ----------
    cfg : ConfigManager or dict
      dict-like with configuration variable name/value-pairs.
    url : str
      URL to be rewritten, if matching configuration is found.

    Returns
    -------
    str
      Rewritten or unmodified URL.
    """
    insteadof = {
        # only leave the base url
        k[4:-10]: v
        for k, v in cfg.items()
        if k.startswith('url.') and k.endswith('.insteadof')
    }

    # all config that applies
    matches = {
        key: v
        for key, val in insteadof.items()
        for v in (val if isinstance(val, tuple) else (val,))
        if url.startswith(v)
    }
    # find longest match, like Git does
    if matches:
        rewrite_base, match = sorted(
            matches.items(),
            key=lambda x: len(x[1]),
            reverse=True,
        )[0]
        if sum(match == v for v in matches.values()) > 1:
            lgr.warning(
                "Ignoring URL rewrite configuration for '%s', "
                "multiple conflicting definitions exists: %s",
                match,
                ['url.{}.insteadof'.format(k)
                 for k, v in matches.items()
                 if v == match]
            )
        else:
            url = '{}{}'.format(rewrite_base, url[len(match):])
    return url


# for convenience, bind to class too
ConfigManager.rewrite_url = rewrite_url

#
# Helpers for bypassing git-config when _writing_ config items,
# mostly useful when a large number of changes needs to be made
# and directly file manipulation without a safety net is worth
# the risk for performance reasons.
#

def quote_config(v):
    """Helper to perform minimal quoting of config keys/value parts

    Parameters
    ----------
    v : str
      To-be-quoted string
    """
    white = (' ', '\t')
    # backslashes need to be quoted in any case
    v = v.replace('\\', '\\\\')
    # must not have additional unquoted quotes
    v = v.replace('"', '\\"')
    if v[0] in white or v[-1] in white:
        # quoting the value due to leading/trailing whitespace
        v = '"{}"'.format(v)
    return v


def write_config_section(fobj, suite, name, props):
    """Write a config section with (multiple) settings.

    Parameters
    ----------
    fobj : File
       Opened target file
    suite : str
       First item of the section name, e.g. 'submodule', or
       'datalad'
    name : str
       Remainder of the section name
    props : dict
       Keys are configuration setting names within the section
       context (i.e. not duplicating `suite` and/or `name`, values
       are configuration setting values.
    """
    fmt = '[{_suite_} {_q_}{_name_}{_q_}]\n'
    for p in props:
        fmt += '\t{p} = {{{p}}}\n'.format(p=p)
    quoted_name = quote_config(name)
    fobj.write(
        fmt.format(
            _suite_=suite,
            _q_='' if quoted_name.startswith('"') else '"',
            _name_=quoted_name,
            **{k: quote_config(v) for k, v in props.items()}))

```
--- END FILE ---

Please first localize the bug based on the issue statement, and then generate *SEARCH/REPLACE* edits to fix the issue.

Every *SEARCH/REPLACE* edit must use this format:
1. The file path
2. The start of search block: <<<<<<< SEARCH
3. A contiguous chunk of lines to search for in the existing source code
4. The dividing line: =======
5. The lines to replace into the source code
6. The end of the replace block: >>>>>>> REPLACE

Here is an example:

```python
### mathweb/flask/app.py
<<<<<<< SEARCH
from flask import Flask
=======
import math
from flask import Flask
>>>>>>> REPLACE
```
</solution>

Please note that the *SEARCH/REPLACE* edit REQUIRES PROPER INDENTATION. If you would like to add the line '        print(x)', you must fully write that out, with all those spaces before the code!
Wrap each *SEARCH/REPLACE* edit in a code block as shown in the example above. If you have multiple *SEARCH/REPLACE* edits, use a separate code block for each one.
[ISSUE]
**Title:** Environment Variable `DATALAD_CONFIG_OVERRIDES` Not Applied, Causing Configuration Key Error

**Description:**
When setting the `DATALAD_CONFIG_OVERRIDES` environment variable with JSON-formatted overrides, the configuration does not apply these overrides correctly. As a result, accessing certain configuration keys raises a `KeyError`, preventing expected configurations from being recognized.

**Example Code:**
```python
import os
from datalad.config import ConfigManager

# Set the environment variable with JSON overrides
os.environ['DATALAD_CONFIG_OVERRIDES'] = '{"datalad.FoO": "val"}'

# Initialize and reload the configuration
cfg = ConfigManager()
cfg.reload()

# Attempt to access the overridden configuration key
print(cfg["datalad.FoO"])  # This raises KeyError
```

**Expected Behavior:**
The configuration key `datalad.FoO` should be set to `"val"` as specified in the `DATALAD_CONFIG_OVERRIDES` environment variable, allowing the program to access it without errors.

**Actual Behavior:**
Accessing `cfg["datalad.FoO"]` raises a `KeyError` because the overrides specified in `DATALAD_CONFIG_OVERRIDES` are not applied correctly.

[/ISSUE]
{
    "test_from_env": "PASSED",
    "test_from_env_overrides": "PASSED",
    "test_overrides": "PASSED",
    "test_rewrite_url": "PASSED",
    "test_procedure_discovery": "ERROR",
    "test_something": "ERROR",
    "test_no_leaks": "ERROR",
    "test_invalid_call": "FAILED",
    "test_dirty": "FAILED",
    "test_configs": "FAILED",
    "test_spaces": "FAILED",
    "test_quoting": "FAILED",
    "test_text2git": "FAILED",
    "test_name_with_underscore": "FAILED",
    "test_crazy_cfg": "FAILED",
    "test_obtain": "FAILED",
    "test_no_local_write_if_no_dataset": "FAILED",
    "test_dataset_local_mode": "FAILED",
    "test_dataset_systemglobal_mode": "FAILED",
    "test_global_config": "FAILED",
    "test_bare": "FAILED",
    "test_write_config_section": "FAILED",
    "test_external_modification": "FAILED"
}
diff --git a/datalad/config.py b/datalad/config.py
index 204eae6fc..6ae65ddc2 100644
--- a/datalad/config.py
+++ b/datalad/config.py
@@ -125,11 +125,11 @@ def _update_from_env(store):
     overrides = {}
     dct = {}
     for k in os.environ:
-        if k == "DATALAD_CONFIG_OVERRIDES":
+        if k == "DATALAD_CONFIG_OVERRIDES_JSON":
             try:
                 overrides = json.loads(os.environ[k])
             except json.decoder.JSONDecodeError as exc:
-                lgr.warning("Failed to load DATALAD_CONFIG_OVERRIDES: %s",
+                lgr.warning("Failed to load DATALAD_CONFIG_OVERRIDES_JSON: %s",
                             exc)
         elif k.startswith('DATALAD_'):
             dct[k.replace('__', '-').replace('_', '.').lower()] = os.environ[k]

{"datalad/config.py": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"\n\"\"\"\n\nfrom collections import namedtuple\n\nimport json\nimport threading\nfrom fasteners import InterProcessLock\nfrom functools import lru_cache\nimport datalad\nfrom datalad.consts import (\n    DATASET_CONFIG_FILE,\n)\nfrom datalad.cmd import (\n    GitWitlessRunner,\n    StdOutErrCapture,\n)\nfrom datalad.dochelpers import exc_str\n\nimport re\nimport os\nfrom pathlib import Path\n\nimport logging\nlgr = logging.getLogger('datalad.config')\n\ncfg_kv_regex = re.compile(r'(^.*)\\n(.*)$', flags=re.MULTILINE)\ncfg_section_regex = re.compile(r'(.*)\\.[^.]+')\ncfg_sectionoption_regex = re.compile(r'(.*)\\.([^.]+)')\n\n\n_where_reload_doc = \"\"\"\n        where : {'dataset', 'local', 'global', 'override'}, optional\n          Indicator which configuration file to modify. 'dataset' indicates the\n          persistent configuration in .datalad/config of a dataset; 'local'\n          the configuration of a dataset's Git repository in .git/config;\n          'global' refers to the general configuration that is not specific to\n          a single repository (usually in $USER/.gitconfig); 'override'\n          limits the modification to the ConfigManager instance, and the\n          assigned value overrides any setting from any other source.\n        reload : bool\n          Flag whether to reload the configuration from file(s) after\n          modification. This can be disable to make multiple sequential\n          modifications slightly more efficient.\"\"\".lstrip()\n\n# Selection of os.stat_result fields we care to collect/compare to judge\n# on either file has changed to warrant reload of configuration.\n_stat_result = namedtuple('_stat_result', 'st_ino st_size st_ctime st_mtime')\n\n\n# we cannot import external_versions here, as the cfg comes before anything\n# and we would have circular imports\n@lru_cache()\ndef get_git_version(runner=None):\n    \"\"\"Return version of available git\"\"\"\n    runner = runner or GitWitlessRunner()\n    return runner.run('git version'.split(),\n                      protocol=StdOutErrCapture)['stdout'].split()[2]\n\n\ndef _where_reload(obj):\n    \"\"\"Helper decorator to simplify providing repetitive docstring\"\"\"\n    obj.__doc__ = obj.__doc__ % _where_reload_doc\n    return obj\n\n\n# TODO document and make \"public\" (used in GitRepo too)\ndef _parse_gitconfig_dump(dump, cwd=None, multi_value=True):\n    \"\"\"Parse a dump-string from `git config -z --list`\n\n    Parameters\n    ----------\n    dump : str\n      Null-byte separated output\n    cwd : path-like, optional\n      Use this path to convert relative paths for origin reports\n      into absolute paths\n    multi_value : bool, optional\n      If True, report values from multiple specifications of the\n      same key as a tuple of values assigned to this key. Otherwise,\n      the last configuration is reported.\n    \"\"\"\n    dct = {}\n    fileset = set()\n    for line in dump.split('\\0'):\n        if not line:\n            continue\n        if line.startswith('file:'):\n            # origin line\n            fname = Path(line[5:])\n            if not fname.is_absolute():\n                fname = Path(cwd) / fname if cwd else Path.cwd() / fname\n            fileset.add(fname)\n            continue\n        if line.startswith('command line:'):\n            # nothing we could handle\n            continue\n        kv_match = cfg_kv_regex.match(line)\n        if kv_match:\n            k, v = kv_match.groups()\n        else:\n            # could be just a key without = value, which git treats as True\n            # if asked for a bool\n            k, v = line, None\n        present_v = dct.get(k, None)\n        if present_v is None or not multi_value:\n            dct[k] = v\n        else:\n            if isinstance(present_v, tuple):\n                dct[k] = present_v + (v,)\n            else:\n                dct[k] = (present_v, v)\n    return dct, fileset\n\n\ndef _update_from_env(store):\n    overrides = {}\n    dct = {}\n    for k in os.environ:\n        if k == \"DATALAD_CONFIG_OVERRIDES\":\n            try:\n                overrides = json.loads(os.environ[k])\n            except json.decoder.JSONDecodeError as exc:\n                lgr.warning(\"Failed to load DATALAD_CONFIG_OVERRIDES: %s\",\n                            exc)\n        elif k.startswith('DATALAD_'):\n            dct[k.replace('__', '-').replace('_', '.').lower()] = os.environ[k]\n    if overrides:\n        store.update(overrides)\n    store.update(dct)\n\n\ndef anything2bool(val):\n    if hasattr(val, 'lower'):\n        val = val.lower()\n    if val in {\"off\", \"no\", \"false\", \"0\"} or not bool(val):\n        return False\n    elif val in {\"on\", \"yes\", \"true\", True} \\\n            or (hasattr(val, 'isdigit') and val.isdigit() and int(val)) \\\n            or isinstance(val, int) and val:\n        return True\n    else:\n        raise TypeError(\n            \"Got value %s which could not be interpreted as a boolean\"\n            % repr(val))\n\n\nclass ConfigManager(object):\n    \"\"\"Thin wrapper around `git-config` with support for a dataset configuration.\n\n    The general idea is to have an object that is primarily used to read/query\n    configuration option.  Upon creation, current configuration is read via one\n    (or max two, in the case of the presence of dataset-specific configuration)\n    calls to `git config`.  If this class is initialized with a Dataset\n    instance, it supports reading and writing configuration from\n    ``.datalad/config`` inside a dataset too. This file is committed to Git and\n    hence useful to ship certain configuration items with a dataset.\n\n    The API aims to provide the most significant read-access API of a\n    dictionary, the Python ConfigParser, and GitPython's config parser\n    implementations.\n\n    This class is presently not capable of efficiently writing multiple\n    configurations items at once.  Instead, each modification results in a\n    dedicated call to `git config`. This author thinks this is OK, as he\n    cannot think of a situation where a large number of items need to be\n    written during normal operation.\n\n    Each instance carries a public `overrides` attribute. This dictionary\n    contains variables that override any setting read from a file. The overrides\n    are persistent across reloads.\n\n    Any DATALAD_* environment variable is also presented as a configuration\n    item. Settings read from environment variables are not stored in any of the\n    configuration files, but are read dynamically from the environment at each\n    `reload()` call. Their values take precedence over any specification in\n    configuration files, and even overrides.\n\n    Parameters\n    ----------\n    dataset : Dataset, optional\n      If provided, all `git config` calls are executed in this dataset's\n      directory. Moreover, any modifications are, by default, directed to\n      this dataset's configuration file (which will be created on demand)\n    overrides : dict, optional\n      Variable overrides, see general class documentation for details.\n    source : {'any', 'local', 'dataset', 'dataset-local'}, optional\n      Which sources of configuration setting to consider. If 'dataset',\n      configuration items are only read from a dataset's persistent\n      configuration file, if any is present (the one in ``.datalad/config``, not\n      ``.git/config``); if 'local', any non-committed source is considered\n      (local and global configuration in Git config's terminology);\n      if 'dataset-local', persistent dataset configuration and local, but\n      not global or system configuration are considered; if 'any'\n      all possible sources of configuration are considered.\n    \"\"\"\n\n    _checked_git_identity = False\n\n    # Lock for running changing operation across multiple threads.\n    # Since config itself to the same path could\n    # potentially be created independently in multiple threads, and we might be\n    # modifying global config as well, making lock static should not allow more than\n    # one thread to  write at a time, even if to different repositories.\n    _run_lock = threading.Lock()\n\n    def __init__(self, dataset=None, overrides=None, source='any'):\n        if source not in ('any', 'local', 'dataset', 'dataset-local'):\n            raise ValueError(\n                'Unknown ConfigManager(source=) setting: {}'.format(source))\n        store = dict(\n            # store in a simple dict\n            # no subclassing, because we want to be largely read-only, and implement\n            # config writing separately\n            cfg={},\n            # track the files that jointly make up the config in this store\n            files=set(),\n            # and their modification times to be able to avoid needless unforced reloads\n            stats=None,\n        )\n        self._stores = dict(\n            # populated with info from git\n            git=store,\n            # only populated with info from commited dataset config\n            dataset=store.copy(),\n        )\n        # merged representation (the only one that existed pre datalad 0.14)\n        # will be built on initial reload\n        self._merged_store = {}\n\n        self._repo_dot_git = None\n        self._repo_pathobj = None\n        if dataset:\n            if hasattr(dataset, 'dot_git'):\n                self._repo_dot_git = dataset.dot_git\n                self._repo_pathobj = dataset.pathobj\n            elif dataset.repo:\n                self._repo_dot_git = dataset.repo.dot_git\n                self._repo_pathobj = dataset.repo.pathobj\n\n        self._config_cmd = ['git', 'config']\n        # public dict to store variables that always override any setting\n        # read from a file\n        # `hasattr()` is needed because `datalad.cfg` is generated upon first module\n        # import, hence when this code runs first, there cannot be any config manager\n        # to inherit from\n        self.overrides = datalad.cfg.overrides.copy() if hasattr(datalad, 'cfg') else {}\n        if overrides is not None:\n            self.overrides.update(overrides)\n        if dataset is None:\n            if source in ('dataset', 'dataset-local'):\n                raise ValueError(\n                    'ConfigManager configured to read dataset only, '\n                    'but no dataset given')\n            # The caller didn't specify a repository. Unset the git directory\n            # when calling 'git config' to prevent a repository in the current\n            # working directory from leaking configuration into the output.\n            self._config_cmd = ['git', '--git-dir=', 'config']\n\n        self._src_mode = source\n        run_kwargs = dict()\n        self._runner = None\n        if dataset is not None:\n            if hasattr(dataset, '_git_runner'):\n                self._runner = dataset._git_runner\n            elif dataset.repo:\n                self._runner = dataset.repo._git_runner\n            else:\n                # make sure we run the git config calls in the dataset\n                # to pick up the right config files\n                run_kwargs['cwd'] = dataset.path\n        if self._runner is None:\n            self._runner = GitWitlessRunner(**run_kwargs)\n\n        self.reload(force=True)\n\n        if not ConfigManager._checked_git_identity:\n            for cfg, envs in (\n                    ('user.name', ('GIT_AUTHOR_NAME', 'GIT_COMMITTER_NAME')),\n                    ('user.email', ('GIT_AUTHOR_EMAIL', 'GIT_COMMITTER_EMAIL'))):\n                if cfg not in self \\\n                        and not any(e in os.environ for e in envs):\n                    lgr.warning(\n                        \"It is highly recommended to configure Git before using \"\n                        \"DataLad. Set both 'user.name' and 'user.email' \"\n                        \"configuration variables.\"\n                    )\n            ConfigManager._checked_git_identity = True\n\n    def reload(self, force=False):\n        \"\"\"Reload all configuration items from the configured sources\n\n        If `force` is False, all files configuration was previously read from\n        are checked for differences in the modification times. If no difference\n        is found for any file no reload is performed. This mechanism will not\n        detect newly created global configuration files, use `force` in this case.\n        \"\"\"\n        run_args = ['-z', '-l', '--show-origin']\n\n        # update from desired config sources only\n        # 2-step strategy:\n        #   - load datalad dataset config from dataset\n        #   - load git config from all supported by git sources\n        # in doing so we always stay compatible with where Git gets its\n        # config from, but also allow to override persistent information\n        # from dataset locally or globally\n\n        # figure out what needs to be reloaded at all\n        to_run = {}\n        # committed dataset config\n        dataset_cfgfile = self._repo_pathobj / DATASET_CONFIG_FILE \\\n            if self._repo_pathobj else None\n        if (self._src_mode != 'local' and\n                dataset_cfgfile and\n                dataset_cfgfile.exists()) and (\n                force or self._need_reload(self._stores['dataset'])):\n            to_run['dataset'] = run_args + ['--file', str(dataset_cfgfile)]\n\n        if self._src_mode != 'dataset' and (\n                force or self._need_reload(self._stores['git'])):\n            to_run['git'] = run_args + ['--local'] \\\n                if self._src_mode == 'dataset-local' \\\n                else run_args\n\n        # reload everything that was found todo\n        while to_run:\n            store_id, runargs = to_run.popitem()\n            self._stores[store_id] = self._reload(runargs)\n\n        # always update the merged representation, even if we did not reload\n        # anything from a file. ENV or overrides could change independently\n        # start with the commit dataset config\n        merged = self._stores['dataset']['cfg'].copy()\n        # local config always takes precedence\n        merged.update(self._stores['git']['cfg'])\n        # superimpose overrides\n        merged.update(self.overrides)\n        # override with environment variables, unless we only want to read the\n        # dataset's commit config\n        if self._src_mode != 'dataset':\n            _update_from_env(merged)\n        self._merged_store = merged\n\n    def _need_reload(self, store):\n        storestats = store['stats']\n        if not storestats:\n            return True\n\n        # we have read files before\n        # check if any file we read from has changed\n        curstats = self._get_stats(store)\n        return any(curstats[f] != storestats[f] for f in store['files'])\n\n    def _reload(self, run_args):\n        # query git-config\n        stdout, stderr = self._run(\n            run_args,\n            protocol=StdOutErrCapture,\n            # always expect git-config to output utf-8\n            encoding='utf-8',\n        )\n        store = {}\n        store['cfg'], store['files'] = _parse_gitconfig_dump(\n            stdout, cwd=self._runner.cwd)\n\n        # update stats of config files, they have just been discovered\n        # and should still exist\n        store['stats'] = self._get_stats(store)\n        return store\n\n    @staticmethod\n    def _get_stats(store):\n        stats = {}\n        for f in store['files']:\n            if f.exists:\n                stat = f.stat()\n                stats[f] = _stat_result(stat.st_ino, stat.st_size, stat.st_ctime, stat.st_mtime)\n            else:\n                stats[f] = None\n        return stats\n\n    @_where_reload\n    def obtain(self, var, default=None, dialog_type=None, valtype=None,\n               store=False, where=None, reload=True, **kwargs):\n        \"\"\"\n        Convenience method to obtain settings interactively, if needed\n\n        A UI will be used to ask for user input in interactive sessions.\n        Questions to ask, and additional explanations can be passed directly\n        as arguments, or retrieved from a list of pre-configured items.\n\n        Additionally, this method allows for type conversion and storage\n        of obtained settings. Both aspects can also be pre-configured.\n\n        Parameters\n        ----------\n        var : str\n          Variable name including any section like `git config` expects them,\n          e.g. 'core.editor'\n        default : any type\n          In interactive sessions and if `store` is True, this default value\n          will be presented to the user for confirmation (or modification).\n          In all other cases, this value will be silently assigned unless\n          there is an existing configuration setting.\n        dialog_type : {'question', 'yesno', None}\n          Which dialog type to use in interactive sessions. If `None`,\n          pre-configured UI options are used.\n        store : bool\n          Whether to store the obtained value (or default)\n        %s\n        `**kwargs`\n          Additional arguments for the UI function call, such as a question\n          `text`.\n        \"\"\"\n        # do local import, as this module is import prominently and the\n        # could theroetically import all kind of weired things for type\n        # conversion\n        from datalad.interface.common_cfg import definitions as cfg_defs\n        # fetch what we know about this variable\n        cdef = cfg_defs.get(var, {})\n        # type conversion setup\n        if valtype is None and 'type' in cdef:\n            valtype = cdef['type']\n        if valtype is None:\n            valtype = lambda x: x\n\n        # any default?\n        if default is None and 'default' in cdef:\n            default = cdef['default']\n\n        _value = None\n        if var in self:\n            # nothing needs to be obtained, it is all here already\n            _value = self[var]\n        elif store is False and default is not None:\n            # nothing will be stored, and we have a default -> no user confirmation\n            # we cannot use logging, because we want to use the config to confiugre\n            # the logging\n            #lgr.debug('using default {} for config setting {}'.format(default, var))\n            _value = default\n\n        if _value is not None:\n            # we got everything we need and can exit early\n            try:\n                return valtype(_value)\n            except Exception as e:\n                raise ValueError(\n                    \"value '{}' of existing configuration for '{}' cannot be \"\n                    \"converted to the desired type '{}' ({})\".format(\n                        _value, var, valtype, exc_str(e)))\n\n        # now we need to try to obtain something from the user\n        from datalad.ui import ui\n\n        # configure UI\n        dialog_opts = kwargs\n        if dialog_type is None:  # no override\n            # check for common knowledge on how to obtain a value\n            if 'ui' in cdef:\n                dialog_type = cdef['ui'][0]\n                # pull standard dialog settings\n                dialog_opts = cdef['ui'][1]\n                # update with input\n                dialog_opts.update(kwargs)\n\n        if (not ui.is_interactive or dialog_type is None) and default is None:\n            raise RuntimeError(\n                \"cannot obtain value for configuration item '{}', \"\n                \"not preconfigured, no default, no UI available\".format(var))\n\n        if not hasattr(ui, dialog_type):\n            raise ValueError(\"UI '{}' does not support dialog type '{}'\".format(\n                ui, dialog_type))\n\n        # configure storage destination, if needed\n        if store:\n            if where is None and 'destination' in cdef:\n                where = cdef['destination']\n            if where is None:\n                raise ValueError(\n                    \"request to store configuration item '{}', but no \"\n                    \"storage destination specified\".format(var))\n\n        # obtain via UI\n        dialog = getattr(ui, dialog_type)\n        _value = dialog(default=default, **dialog_opts)\n\n        if _value is None:\n            # we got nothing\n            if default is None:\n                raise RuntimeError(\n                    \"could not obtain value for configuration item '{}', \"\n                    \"not preconfigured, no default\".format(var))\n            # XXX maybe we should return default here, even it was returned\n            # from the UI -- if that is even possible\n\n        # execute type conversion before storing to check that we got\n        # something that looks like what we want\n        try:\n            value = valtype(_value)\n        except Exception as e:\n            raise ValueError(\n                \"cannot convert user input `{}` to desired type ({})\".format(\n                    _value, exc_str(e)))\n            # XXX we could consider \"looping\" until we have a value of proper\n            # type in case of a user typo...\n\n        if store:\n            # store value as it was before any conversion, needs to be str\n            # anyway\n            # needs string conversion nevertheless, because default could come\n            # in as something else\n            self.add(var, '{}'.format(_value), where=where, reload=reload)\n        return value\n\n    def __repr__(self):\n        # give full list of all tracked config files, plus overrides\n        return \"ConfigManager({}{})\".format(\n            [str(p) for p in self._stores['dataset']['files'].union(\n                self._stores['git']['files'])],\n            ', overrides={!r}'.format(self.overrides) if self.overrides else '',\n        )\n\n    def __str__(self):\n        # give path of dataset, if there is any, plus overrides\n        return \"ConfigManager({}{})\".format(\n            self._repo_pathobj if self._repo_pathobj else '',\n            'with overrides' if self.overrides else '',\n        )\n\n    #\n    # Compatibility with dict API\n    #\n    def __len__(self):\n        return len(self._merged_store)\n\n    def __getitem__(self, key):\n        return self._merged_store.__getitem__(key)\n\n    def __contains__(self, key):\n        return self._merged_store.__contains__(key)\n\n    def keys(self):\n        \"\"\"Returns list of configuration item names\"\"\"\n        return self._merged_store.keys()\n\n    # XXX should this be *args?\n    def get(self, key, default=None, get_all=False):\n        \"\"\"D.get(k[,d]) -> D[k] if k in D, else d.  d defaults to None.\n\n        Parameters\n        ----------\n        default : optional\n          Value to return when key is not present. `None` by default.\n        get_all : bool, optional\n          If True, return all values of multiple identical configuration keys.\n          By default only the last specified value is returned.\n        \"\"\"\n        try:\n            val = self[key]\n            if get_all or not isinstance(val, tuple):\n                return val\n            else:\n                return val[-1]\n        except KeyError:\n            # return as-is, default could be a tuple, hence do not subject to\n            # get_all processing\n            return default\n\n    def get_from_source(self, source, key, default=None):\n        \"\"\"Like get(), but a source can be specific.\n\n        If `source` is 'dataset', only the commited configuration is queried,\n        overrides are applied. In the case of 'local', the committed\n        configuration is ignored, but overrides and configuration from\n        environment variables are applied as usual.\n        \"\"\"\n        if source not in ('dataset', 'local'):\n            raise ValueError(\"source must be 'dataset' or 'local'\")\n        if source == 'dataset':\n            return self.overrides.get(\n                key,\n                self._stores['dataset']['cfg'].get(\n                    key,\n                    default))\n        else:\n            if key not in self._stores['dataset']['cfg']:\n                # the key is not in the committed config, hence we can\n                # just report based on the merged representation\n                return self.get(key, default)\n            else:\n                # expensive case, rebuild a config without the committed\n                # dataset config contributing\n                env = {}\n                _update_from_env(env)\n                return env.get(\n                    key,\n                    self.overrides.get(\n                        key,\n                        self._stores['local']['cfg'].get(\n                            key,\n                            default)))\n\n    #\n    # Compatibility with ConfigParser API\n    #\n    def sections(self):\n        \"\"\"Returns a list of the sections available\"\"\"\n        return list(set([cfg_section_regex.match(k).group(1) for k in self._merged_store]))\n\n    def options(self, section):\n        \"\"\"Returns a list of options available in the specified section.\"\"\"\n        opts = []\n        for k in self._merged_store:\n            sec, opt = cfg_sectionoption_regex.match(k).groups()\n            if sec == section:\n                opts.append(opt)\n        return opts\n\n    def has_section(self, section):\n        \"\"\"Indicates whether a section is present in the configuration\"\"\"\n        for k in self._merged_store:\n            if k.startswith(section):\n                return True\n        return False\n\n    def has_option(self, section, option):\n        \"\"\"If the given section exists, and contains the given option\"\"\"\n        for k in self._merged_store:\n            sec, opt = cfg_sectionoption_regex.match(k).groups()\n            if sec == section and opt == option:\n                return True\n        return False\n\n    def _get_type(self, typefn, section, option):\n        key = '.'.join([section, option])\n        # Mimic the handling of get_value(..., default=None), while still going\n        # through get() in order to get its default tuple handling.\n        if key not in self:\n            raise KeyError(key)\n        return typefn(self.get(key))\n\n    def getint(self, section, option):\n        \"\"\"A convenience method which coerces the option value to an integer\"\"\"\n        return self._get_type(int, section, option)\n\n    def getfloat(self, section, option):\n        \"\"\"A convenience method which coerces the option value to a float\"\"\"\n        return self._get_type(float, section, option)\n\n    def getbool(self, section, option, default=None):\n        \"\"\"A convenience method which coerces the option value to a bool\n\n        Values \"on\", \"yes\", \"true\" and any int!=0 are considered True\n        Values which evaluate to bool False, \"off\", \"no\", \"false\" are considered\n        False\n        TypeError is raised for other values.\n        \"\"\"\n        key = '.'.join([section, option])\n        # Mimic the handling of get_value(..., default=None), while still going\n        # through get() in order to get its default tuple handling.\n        if default is None and key not in self:\n            raise KeyError(key)\n        val = self.get(key, default=default)\n        if val is None:  # no value at all, git treats it as True\n            return True\n        return anything2bool(val)\n\n    # this is a hybrid of ConfigParser and dict API\n    def items(self, section=None):\n        \"\"\"Return a list of (name, value) pairs for each option\n\n        Optionally limited to a given section.\n        \"\"\"\n        if section is None:\n            return self._merged_store.items()\n        return [(k, v) for k, v in self._merged_store.items()\n                if cfg_section_regex.match(k).group(1) == section]\n\n    #\n    # Compatibility with GitPython's ConfigParser\n    #\n    def get_value(self, section, option, default=None):\n        \"\"\"Like `get()`, but with an optional default value\n\n        If the default is not None, the given default value will be returned in\n        case the option did not exist. This behavior imitates GitPython's\n        config parser.\n        \"\"\"\n        try:\n            return self['.'.join((section, option))]\n        except KeyError as e:\n            # this strange dance is needed because gitpython does it this way\n            if default is not None:\n                return default\n            else:\n                raise e\n\n    #\n    # Modify configuration (proxy respective git-config call)\n    #\n    @_where_reload\n    def _run(self, args, where=None, reload=False, **kwargs):\n        \"\"\"Centralized helper to run \"git config\" calls\n\n        Parameters\n        ----------\n        args : list\n          Arguments to pass for git config\n        %s\n        **kwargs\n          Keywords arguments for Runner's call\n        \"\"\"\n        if where:\n            args = self._get_location_args(where) + args\n        if '-l' in args:\n            # we are just reading, no need to reload, no need to lock\n            out = self._runner.run(self._config_cmd + args, **kwargs)\n            return out['stdout'], out['stderr']\n\n        # all other calls are modifications\n        if '--file' in args:\n            # all paths we are passing are absolute\n            custom_file = Path(args[args.index('--file') + 1])\n            custom_file.parent.mkdir(exist_ok=True)\n        lockfile = None\n        if self._repo_dot_git and ('--local' in args or '--file' in args):\n            # modification of config in a dataset\n            lockfile = self._repo_dot_git / 'config.dataladlock'\n        else:\n            # follow pattern in downloaders for lockfile location\n            lockfile = Path(self.obtain('datalad.locations.cache')) \\\n                / 'locks' / 'gitconfig.lck'\n\n        with ConfigManager._run_lock, InterProcessLock(lockfile, logger=lgr):\n            out = self._runner.run(self._config_cmd + args, **kwargs)\n\n        if reload:\n            self.reload()\n        return out['stdout'], out['stderr']\n\n    def _get_location_args(self, where, args=None):\n        if args is None:\n            args = []\n        cfg_labels = ('dataset', 'local', 'global', 'override')\n        if where not in cfg_labels:\n            raise ValueError(\n                \"unknown configuration label '{}' (not in {})\".format(\n                    where, cfg_labels))\n        if where == 'dataset':\n            if not self._repo_pathobj:\n                raise ValueError(\n                    'ConfigManager cannot store configuration to dataset, '\n                    'none specified')\n            dataset_cfgfile = self._repo_pathobj / DATASET_CONFIG_FILE\n            args.extend(['--file', str(dataset_cfgfile)])\n        elif where == 'global':\n            args.append('--global')\n        elif where == 'local':\n            args.append('--local')\n        return args\n\n    @_where_reload\n    def add(self, var, value, where='dataset', reload=True):\n        \"\"\"Add a configuration variable and value\n\n        Parameters\n        ----------\n        var : str\n          Variable name including any section like `git config` expects them, e.g.\n          'core.editor'\n        value : str\n          Variable value\n        %s\"\"\"\n        if where == 'override':\n            from datalad.utils import ensure_list\n            val = ensure_list(self.overrides.pop(var, None))\n            val.append(value)\n            self.overrides[var] = val[0] if len(val) == 1 else val\n            if reload:\n                self.reload(force=True)\n            return\n\n        self._run(['--add', var, value], where=where, reload=reload,\n                  protocol=StdOutErrCapture)\n\n    @_where_reload\n    def set(self, var, value, where='dataset', reload=True, force=False):\n        \"\"\"Set a variable to a value.\n\n        In opposition to `add`, this replaces the value of `var` if there is\n        one already.\n\n        Parameters\n        ----------\n        var : str\n          Variable name including any section like `git config` expects them, e.g.\n          'core.editor'\n        value : str\n          Variable value\n        force: bool\n          if set, replaces all occurrences of `var` by a single one with the\n          given `value`. Otherwise raise if multiple entries for `var` exist\n          already\n        %s\"\"\"\n        if where == 'override':\n            self.overrides[var] = value\n            if reload:\n                self.reload(force=True)\n            return\n\n        from datalad.support.gitrepo import to_options\n\n        self._run(to_options(replace_all=force) + [var, value],\n                  where=where, reload=reload, protocol=StdOutErrCapture)\n\n    @_where_reload\n    def rename_section(self, old, new, where='dataset', reload=True):\n        \"\"\"Rename a configuration section\n\n        Parameters\n        ----------\n        old : str\n          Name of the section to rename.\n        new : str\n          Name of the section to rename to.\n        %s\"\"\"\n        if where == 'override':\n            self.overrides = {\n                (new + k[len(old):]) if k.startswith(old + '.') else k: v\n                for k, v in self.overrides.items()\n            }\n            if reload:\n                self.reload(force=True)\n            return\n\n        self._run(['--rename-section', old, new], where=where, reload=reload)\n\n    @_where_reload\n    def remove_section(self, sec, where='dataset', reload=True):\n        \"\"\"Rename a configuration section\n\n        Parameters\n        ----------\n        sec : str\n          Name of the section to remove.\n        %s\"\"\"\n        if where == 'override':\n            self.overrides = {\n                k: v\n                for k, v in self.overrides.items()\n                if not k.startswith(sec + '.')\n            }\n            if reload:\n                self.reload(force=True)\n            return\n\n        self._run(['--remove-section', sec], where=where, reload=reload)\n\n    @_where_reload\n    def unset(self, var, where='dataset', reload=True):\n        \"\"\"Remove all occurrences of a variable\n\n        Parameters\n        ----------\n        var : str\n          Name of the variable to remove\n        %s\"\"\"\n        if where == 'override':\n            self.overrides.pop(var, None)\n            if reload:\n                self.reload(force=True)\n            return\n\n        # use unset all as it is simpler for now\n        self._run(['--unset-all', var], where=where, reload=reload)\n\n\ndef rewrite_url(cfg, url):\n    \"\"\"Any matching 'url.<base>.insteadOf' configuration is applied\n\n    Any URL that starts with such a configuration will be rewritten\n    to start, instead, with <base>. When more than one insteadOf\n    strings match a given URL, the longest match is used.\n\n    Parameters\n    ----------\n    cfg : ConfigManager or dict\n      dict-like with configuration variable name/value-pairs.\n    url : str\n      URL to be rewritten, if matching configuration is found.\n\n    Returns\n    -------\n    str\n      Rewritten or unmodified URL.\n    \"\"\"\n    insteadof = {\n        # only leave the base url\n        k[4:-10]: v\n        for k, v in cfg.items()\n        if k.startswith('url.') and k.endswith('.insteadof')\n    }\n\n    # all config that applies\n    matches = {\n        key: v\n        for key, val in insteadof.items()\n        for v in (val if isinstance(val, tuple) else (val,))\n        if url.startswith(v)\n    }\n    # find longest match, like Git does\n    if matches:\n        rewrite_base, match = sorted(\n            matches.items(),\n            key=lambda x: len(x[1]),\n            reverse=True,\n        )[0]\n        if sum(match == v for v in matches.values()) > 1:\n            lgr.warning(\n                \"Ignoring URL rewrite configuration for '%s', \"\n                \"multiple conflicting definitions exists: %s\",\n                match,\n                ['url.{}.insteadof'.format(k)\n                 for k, v in matches.items()\n                 if v == match]\n            )\n        else:\n            url = '{}{}'.format(rewrite_base, url[len(match):])\n    return url\n\n\n# for convenience, bind to class too\nConfigManager.rewrite_url = rewrite_url\n\n#\n# Helpers for bypassing git-config when _writing_ config items,\n# mostly useful when a large number of changes needs to be made\n# and directly file manipulation without a safety net is worth\n# the risk for performance reasons.\n#\n\ndef quote_config(v):\n    \"\"\"Helper to perform minimal quoting of config keys/value parts\n\n    Parameters\n    ----------\n    v : str\n      To-be-quoted string\n    \"\"\"\n    white = (' ', '\\t')\n    # backslashes need to be quoted in any case\n    v = v.replace('\\\\', '\\\\\\\\')\n    # must not have additional unquoted quotes\n    v = v.replace('\"', '\\\\\"')\n    if v[0] in white or v[-1] in white:\n        # quoting the value due to leading/trailing whitespace\n        v = '\"{}\"'.format(v)\n    return v\n\n\ndef write_config_section(fobj, suite, name, props):\n    \"\"\"Write a config section with (multiple) settings.\n\n    Parameters\n    ----------\n    fobj : File\n       Opened target file\n    suite : str\n       First item of the section name, e.g. 'submodule', or\n       'datalad'\n    name : str\n       Remainder of the section name\n    props : dict\n       Keys are configuration setting names within the section\n       context (i.e. not duplicating `suite` and/or `name`, values\n       are configuration setting values.\n    \"\"\"\n    fmt = '[{_suite_} {_q_}{_name_}{_q_}]\\n'\n    for p in props:\n        fmt += '\\t{p} = {{{p}}}\\n'.format(p=p)\n    quoted_name = quote_config(name)\n    fobj.write(\n        fmt.format(\n            _suite_=suite,\n            _q_='' if quoted_name.startswith('\"') else '\"',\n            _name_=quoted_name,\n            **{k: quote_config(v) for k, v in props.items()}))\n"}
A user will ask you to solve a task. You should first draft your thinking process (inner monologue). Then, generate the solution.

Your response format must follow the template below:
<think>
Your thoughts or/and draft, like working through an exercise on scratch paper. Be as casual and as long as you want until you are confident to generate a correct solution.
</think>
<solution>
Final solution presented to the user.
</solution>
system
We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
[ISSUE]
**Title:** Environment Variable `DATALAD_CONFIG_OVERRIDES` Not Applied, Causing Configuration Key Error

**Description:**
When setting the `DATALAD_CONFIG_OVERRIDES` environment variable with JSON-formatted overrides, the configuration does not apply these overrides correctly. As a result, accessing certain configuration keys raises a `KeyError`, preventing expected configurations from being recognized.

**Example Code:**
```python
import os
from datalad.config import ConfigManager

# Set the environment variable with JSON overrides
os.environ['DATALAD_CONFIG_OVERRIDES'] = '{"datalad.FoO": "val"}'

# Initialize and reload the configuration
cfg = ConfigManager()
cfg.reload()

# Attempt to access the overridden configuration key
print(cfg["datalad.FoO"])  # This raises KeyError
```

**Expected Behavior:**
The configuration key `datalad.FoO` should be set to `"val"` as specified in the `DATALAD_CONFIG_OVERRIDES` environment variable, allowing the program to access it without errors.

**Actual Behavior:**
Accessing `cfg["datalad.FoO"]` raises a `KeyError` because the overrides specified in `DATALAD_CONFIG_OVERRIDES` are not applied correctly.

[/ISSUE]
--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.

--- BEGIN FILE ---
```
### datalad/config.py
# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-
# ex: set sts=4 ts=4 sw=4 noet:
# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
#
#   See COPYING file distributed along with the datalad package for the
#   copyright and license terms.
#
# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
"""
"""

from collections import namedtuple

import json
import threading
from fasteners import InterProcessLock
from functools import lru_cache
import datalad
from datalad.consts import (
    DATASET_CONFIG_FILE,
)
from datalad.cmd import (
    GitWitlessRunner,
    StdOutErrCapture,
)
from datalad.dochelpers import exc_str

import re
import os
from pathlib import Path

import logging
lgr = logging.getLogger('datalad.config')

cfg_kv_regex = re.compile(r'(^.*)\n(.*)$', flags=re.MULTILINE)
cfg_section_regex = re.compile(r'(.*)\.[^.]+')
cfg_sectionoption_regex = re.compile(r'(.*)\.([^.]+)')


_where_reload_doc = """
        where : {'dataset', 'local', 'global', 'override'}, optional
          Indicator which configuration file to modify. 'dataset' indicates the
          persistent configuration in .datalad/config of a dataset; 'local'
          the configuration of a dataset's Git repository in .git/config;
          'global' refers to the general configuration that is not specific to
          a single repository (usually in $USER/.gitconfig); 'override'
          limits the modification to the ConfigManager instance, and the
          assigned value overrides any setting from any other source.
        reload : bool
          Flag whether to reload the configuration from file(s) after
          modification. This can be disable to make multiple sequential
          modifications slightly more efficient.""".lstrip()

# Selection of os.stat_result fields we care to collect/compare to judge
# on either file has changed to warrant reload of configuration.
_stat_result = namedtuple('_stat_result', 'st_ino st_size st_ctime st_mtime')


# we cannot import external_versions here, as the cfg comes before anything
# and we would have circular imports
@lru_cache()
def get_git_version(runner=None):
    """Return version of available git"""
    runner = runner or GitWitlessRunner()
    return runner.run('git version'.split(),
                      protocol=StdOutErrCapture)['stdout'].split()[2]


def _where_reload(obj):
    """Helper decorator to simplify providing repetitive docstring"""
    obj.__doc__ = obj.__doc__ % _where_reload_doc
    return obj


# TODO document and make "public" (used in GitRepo too)
def _parse_gitconfig_dump(dump, cwd=None, multi_value=True):
    """Parse a dump-string from `git config -z --list`

    Parameters
    ----------
    dump : str
      Null-byte separated output
    cwd : path-like, optional
      Use this path to convert relative paths for origin reports
      into absolute paths
    multi_value : bool, optional
      If True, report values from multiple specifications of the
      same key as a tuple of values assigned to this key. Otherwise,
      the last configuration is reported.
    """
    dct = {}
    fileset = set()
    for line in dump.split('\0'):
        if not line:
            continue
        if line.startswith('file:'):
            # origin line
            fname = Path(line[5:])
            if not fname.is_absolute():
                fname = Path(cwd) / fname if cwd else Path.cwd() / fname
            fileset.add(fname)
            continue
        if line.startswith('command line:'):
            # nothing we could handle
            continue
        kv_match = cfg_kv_regex.match(line)
        if kv_match:
            k, v = kv_match.groups()
        else:
            # could be just a key without = value, which git treats as True
            # if asked for a bool
            k, v = line, None
        present_v = dct.get(k, None)
        if present_v is None or not multi_value:
            dct[k] = v
        else:
            if isinstance(present_v, tuple):
                dct[k] = present_v + (v,)
            else:
                dct[k] = (present_v, v)
    return dct, fileset


def _update_from_env(store):
    overrides = {}
    dct = {}
    for k in os.environ:
        if k == "DATALAD_CONFIG_OVERRIDES":
            try:
                overrides = json.loads(os.environ[k])
            except json.decoder.JSONDecodeError as exc:
                lgr.warning("Failed to load DATALAD_CONFIG_OVERRIDES: %s",
                            exc)
        elif k.startswith('DATALAD_'):
            dct[k.replace('__', '-').replace('_', '.').lower()] = os.environ[k]
    if overrides:
        store.update(overrides)
    store.update(dct)


def anything2bool(val):
    if hasattr(val, 'lower'):
        val = val.lower()
    if val in {"off", "no", "false", "0"} or not bool(val):
        return False
    elif val in {"on", "yes", "true", True} \
            or (hasattr(val, 'isdigit') and val.isdigit() and int(val)) \
            or isinstance(val, int) and val:
        return True
    else:
        raise TypeError(
            "Got value %s which could not be interpreted as a boolean"
            % repr(val))


class ConfigManager(object):
    """Thin wrapper around `git-config` with support for a dataset configuration.

    The general idea is to have an object that is primarily used to read/query
    configuration option.  Upon creation, current configuration is read via one
    (or max two, in the case of the presence of dataset-specific configuration)
    calls to `git config`.  If this class is initialized with a Dataset
    instance, it supports reading and writing configuration from
    ``.datalad/config`` inside a dataset too. This file is committed to Git and
    hence useful to ship certain configuration items with a dataset.

    The API aims to provide the most significant read-access API of a
    dictionary, the Python ConfigParser, and GitPython's config parser
    implementations.

    This class is presently not capable of efficiently writing multiple
    configurations items at once.  Instead, each modification results in a
    dedicated call to `git config`. This author thinks this is OK, as he
    cannot think of a situation where a large number of items need to be
    written during normal operation.

    Each instance carries a public `overrides` attribute. This dictionary
    contains variables that override any setting read from a file. The overrides
    are persistent across reloads.

    Any DATALAD_* environment variable is also presented as a configuration
    item. Settings read from environment variables are not stored in any of the
    configuration files, but are read dynamically from the environment at each
    `reload()` call. Their values take precedence over any specification in
    configuration files, and even overrides.

    Parameters
    ----------
    dataset : Dataset, optional
      If provided, all `git config` calls are executed in this dataset's
      directory. Moreover, any modifications are, by default, directed to
      this dataset's configuration file (which will be created on demand)
    overrides : dict, optional
      Variable overrides, see general class documentation for details.
    source : {'any', 'local', 'dataset', 'dataset-local'}, optional
      Which sources of configuration setting to consider. If 'dataset',
      configuration items are only read from a dataset's persistent
      configuration file, if any is present (the one in ``.datalad/config``, not
      ``.git/config``); if 'local', any non-committed source is considered
      (local and global configuration in Git config's terminology);
      if 'dataset-local', persistent dataset configuration and local, but
      not global or system configuration are considered; if 'any'
      all possible sources of configuration are considered.
    """

    _checked_git_identity = False

    # Lock for running changing operation across multiple threads.
    # Since config itself to the same path could
    # potentially be created independently in multiple threads, and we might be
    # modifying global config as well, making lock static should not allow more than
    # one thread to  write at a time, even if to different repositories.
    _run_lock = threading.Lock()

    def __init__(self, dataset=None, overrides=None, source='any'):
        if source not in ('any', 'local', 'dataset', 'dataset-local'):
            raise ValueError(
                'Unknown ConfigManager(source=) setting: {}'.format(source))
        store = dict(
            # store in a simple dict
            # no subclassing, because we want to be largely read-only, and implement
            # config writing separately
            cfg={},
            # track the files that jointly make up the config in this store
            files=set(),
            # and their modification times to be able to avoid needless unforced reloads
            stats=None,
        )
        self._stores = dict(
            # populated with info from git
            git=store,
            # only populated with info from commited dataset config
            dataset=store.copy(),
        )
        # merged representation (the only one that existed pre datalad 0.14)
        # will be built on initial reload
        self._merged_store = {}

        self._repo_dot_git = None
        self._repo_pathobj = None
        if dataset:
            if hasattr(dataset, 'dot_git'):
                self._repo_dot_git = dataset.dot_git
                self._repo_pathobj = dataset.pathobj
            elif dataset.repo:
                self._repo_dot_git = dataset.repo.dot_git
                self._repo_pathobj = dataset.repo.pathobj

        self._config_cmd = ['git', 'config']
        # public dict to store variables that always override any setting
        # read from a file
        # `hasattr()` is needed because `datalad.cfg` is generated upon first module
        # import, hence when this code runs first, there cannot be any config manager
        # to inherit from
        self.overrides = datalad.cfg.overrides.copy() if hasattr(datalad, 'cfg') else {}
        if overrides is not None:
            self.overrides.update(overrides)
        if dataset is None:
            if source in ('dataset', 'dataset-local'):
                raise ValueError(
                    'ConfigManager configured to read dataset only, '
                    'but no dataset given')
            # The caller didn't specify a repository. Unset the git directory
            # when calling 'git config' to prevent a repository in the current
            # working directory from leaking configuration into the output.
            self._config_cmd = ['git', '--git-dir=', 'config']

        self._src_mode = source
        run_kwargs = dict()
        self._runner = None
        if dataset is not None:
            if hasattr(dataset, '_git_runner'):
                self._runner = dataset._git_runner
            elif dataset.repo:
                self._runner = dataset.repo._git_runner
            else:
                # make sure we run the git config calls in the dataset
                # to pick up the right config files
                run_kwargs['cwd'] = dataset.path
        if self._runner is None:
            self._runner = GitWitlessRunner(**run_kwargs)

        self.reload(force=True)

        if not ConfigManager._checked_git_identity:
            for cfg, envs in (
                    ('user.name', ('GIT_AUTHOR_NAME', 'GIT_COMMITTER_NAME')),
                    ('user.email', ('GIT_AUTHOR_EMAIL', 'GIT_COMMITTER_EMAIL'))):
                if cfg not in self \
                        and not any(e in os.environ for e in envs):
                    lgr.warning(
                        "It is highly recommended to configure Git before using "
                        "DataLad. Set both 'user.name' and 'user.email' "
                        "configuration variables."
                    )
            ConfigManager._checked_git_identity = True

    def reload(self, force=False):
        """Reload all configuration items from the configured sources

        If `force` is False, all files configuration was previously read from
        are checked for differences in the modification times. If no difference
        is found for any file no reload is performed. This mechanism will not
        detect newly created global configuration files, use `force` in this case.
        """
        run_args = ['-z', '-l', '--show-origin']

        # update from desired config sources only
        # 2-step strategy:
        #   - load datalad dataset config from dataset
        #   - load git config from all supported by git sources
        # in doing so we always stay compatible with where Git gets its
        # config from, but also allow to override persistent information
        # from dataset locally or globally

        # figure out what needs to be reloaded at all
        to_run = {}
        # committed dataset config
        dataset_cfgfile = self._repo_pathobj / DATASET_CONFIG_FILE \
            if self._repo_pathobj else None
        if (self._src_mode != 'local' and
                dataset_cfgfile and
                dataset_cfgfile.exists()) and (
                force or self._need_reload(self._stores['dataset'])):
            to_run['dataset'] = run_args + ['--file', str(dataset_cfgfile)]

        if self._src_mode != 'dataset' and (
                force or self._need_reload(self._stores['git'])):
            to_run['git'] = run_args + ['--local'] \
                if self._src_mode == 'dataset-local' \
                else run_args

        # reload everything that was found todo
        while to_run:
            store_id, runargs = to_run.popitem()
            self._stores[store_id] = self._reload(runargs)

        # always update the merged representation, even if we did not reload
        # anything from a file. ENV or overrides could change independently
        # start with the commit dataset config
        merged = self._stores['dataset']['cfg'].copy()
        # local config always takes precedence
        merged.update(self._stores['git']['cfg'])
        # superimpose overrides
        merged.update(self.overrides)
        # override with environment variables, unless we only want to read the
        # dataset's commit config
        if self._src_mode != 'dataset':
            _update_from_env(merged)
        self._merged_store = merged

    def _need_reload(self, store):
        storestats = store['stats']
        if not storestats:
            return True

        # we have read files before
        # check if any file we read from has changed
        curstats = self._get_stats(store)
        return any(curstats[f] != storestats[f] for f in store['files'])

    def _reload(self, run_args):
        # query git-config
        stdout, stderr = self._run(
            run_args,
            protocol=StdOutErrCapture,
            # always expect git-config to output utf-8
            encoding='utf-8',
        )
        store = {}
        store['cfg'], store['files'] = _parse_gitconfig_dump(
            stdout, cwd=self._runner.cwd)

        # update stats of config files, they have just been discovered
        # and should still exist
        store['stats'] = self._get_stats(store)
        return store

    @staticmethod
    def _get_stats(store):
        stats = {}
        for f in store['files']:
            if f.exists:
                stat = f.stat()
                stats[f] = _stat_result(stat.st_ino, stat.st_size, stat.st_ctime, stat.st_mtime)
            else:
                stats[f] = None
        return stats

    @_where_reload
    def obtain(self, var, default=None, dialog_type=None, valtype=None,
               store=False, where=None, reload=True, **kwargs):
        """
        Convenience method to obtain settings interactively, if needed

        A UI will be used to ask for user input in interactive sessions.
        Questions to ask, and additional explanations can be passed directly
        as arguments, or retrieved from a list of pre-configured items.

        Additionally, this method allows for type conversion and storage
        of obtained settings. Both aspects can also be pre-configured.

        Parameters
        ----------
        var : str
          Variable name including any section like `git config` expects them,
          e.g. 'core.editor'
        default : any type
          In interactive sessions and if `store` is True, this default value
          will be presented to the user for confirmation (or modification).
          In all other cases, this value will be silently assigned unless
          there is an existing configuration setting.
        dialog_type : {'question', 'yesno', None}
          Which dialog type to use in interactive sessions. If `None`,
          pre-configured UI options are used.
        store : bool
          Whether to store the obtained value (or default)
        %s
        `**kwargs`
          Additional arguments for the UI function call, such as a question
          `text`.
        """
        # do local import, as this module is import prominently and the
        # could theroetically import all kind of weired things for type
        # conversion
        from datalad.interface.common_cfg import definitions as cfg_defs
        # fetch what we know about this variable
        cdef = cfg_defs.get(var, {})
        # type conversion setup
        if valtype is None and 'type' in cdef:
            valtype = cdef['type']
        if valtype is None:
            valtype = lambda x: x

        # any default?
        if default is None and 'default' in cdef:
            default = cdef['default']

        _value = None
        if var in self:
            # nothing needs to be obtained, it is all here already
            _value = self[var]
        elif store is False and default is not None:
            # nothing will be stored, and we have a default -> no user confirmation
            # we cannot use logging, because we want to use the config to confiugre
            # the logging
            #lgr.debug('using default {} for config setting {}'.format(default, var))
            _value = default

        if _value is not None:
            # we got everything we need and can exit early
            try:
                return valtype(_value)
            except Exception as e:
                raise ValueError(
                    "value '{}' of existing configuration for '{}' cannot be "
                    "converted to the desired type '{}' ({})".format(
                        _value, var, valtype, exc_str(e)))

        # now we need to try to obtain something from the user
        from datalad.ui import ui

        # configure UI
        dialog_opts = kwargs
        if dialog_type is None:  # no override
            # check for common knowledge on how to obtain a value
            if 'ui' in cdef:
                dialog_type = cdef['ui'][0]
                # pull standard dialog settings
                dialog_opts = cdef['ui'][1]
                # update with input
                dialog_opts.update(kwargs)

        if (not ui.is_interactive or dialog_type is None) and default is None:
            raise RuntimeError(
                "cannot obtain value for configuration item '{}', "
                "not preconfigured, no default, no UI available".format(var))

        if not hasattr(ui, dialog_type):
            raise ValueError("UI '{}' does not support dialog type '{}'".format(
                ui, dialog_type))

        # configure storage destination, if needed
        if store:
            if where is None and 'destination' in cdef:
                where = cdef['destination']
            if where is None:
                raise ValueError(
                    "request to store configuration item '{}', but no "
                    "storage destination specified".format(var))

        # obtain via UI
        dialog = getattr(ui, dialog_type)
        _value = dialog(default=default, **dialog_opts)

        if _value is None:
            # we got nothing
            if default is None:
                raise RuntimeError(
                    "could not obtain value for configuration item '{}', "
                    "not preconfigured, no default".format(var))
            # XXX maybe we should return default here, even it was returned
            # from the UI -- if that is even possible

        # execute type conversion before storing to check that we got
        # something that looks like what we want
        try:
            value = valtype(_value)
        except Exception as e:
            raise ValueError(
                "cannot convert user input `{}` to desired type ({})".format(
                    _value, exc_str(e)))
            # XXX we could consider "looping" until we have a value of proper
            # type in case of a user typo...

        if store:
            # store value as it was before any conversion, needs to be str
            # anyway
            # needs string conversion nevertheless, because default could come
            # in as something else
            self.add(var, '{}'.format(_value), where=where, reload=reload)
        return value

    def __repr__(self):
        # give full list of all tracked config files, plus overrides
        return "ConfigManager({}{})".format(
            [str(p) for p in self._stores['dataset']['files'].union(
                self._stores['git']['files'])],
            ', overrides={!r}'.format(self.overrides) if self.overrides else '',
        )

    def __str__(self):
        # give path of dataset, if there is any, plus overrides
        return "ConfigManager({}{})".format(
            self._repo_pathobj if self._repo_pathobj else '',
            'with overrides' if self.overrides else '',
        )

    #
    # Compatibility with dict API
    #
    def __len__(self):
        return len(self._merged_store)

    def __getitem__(self, key):
        return self._merged_store.__getitem__(key)

    def __contains__(self, key):
        return self._merged_store.__contains__(key)

    def keys(self):
        """Returns list of configuration item names"""
        return self._merged_store.keys()

    # XXX should this be *args?
    def get(self, key, default=None, get_all=False):
        """D.get(k[,d]) -> D[k] if k in D, else d.  d defaults to None.

        Parameters
        ----------
        default : optional
          Value to return when key is not present. `None` by default.
        get_all : bool, optional
          If True, return all values of multiple identical configuration keys.
          By default only the last specified value is returned.
        """
        try:
            val = self[key]
            if get_all or not isinstance(val, tuple):
                return val
            else:
                return val[-1]
        except KeyError:
            # return as-is, default could be a tuple, hence do not subject to
            # get_all processing
            return default

    def get_from_source(self, source, key, default=None):
        """Like get(), but a source can be specific.

        If `source` is 'dataset', only the commited configuration is queried,
        overrides are applied. In the case of 'local', the committed
        configuration is ignored, but overrides and configuration from
        environment variables are applied as usual.
        """
        if source not in ('dataset', 'local'):
            raise ValueError("source must be 'dataset' or 'local'")
        if source == 'dataset':
            return self.overrides.get(
                key,
                self._stores['dataset']['cfg'].get(
                    key,
                    default))
        else:
            if key not in self._stores['dataset']['cfg']:
                # the key is not in the committed config, hence we can
                # just report based on the merged representation
                return self.get(key, default)
            else:
                # expensive case, rebuild a config without the committed
                # dataset config contributing
                env = {}
                _update_from_env(env)
                return env.get(
                    key,
                    self.overrides.get(
                        key,
                        self._stores['local']['cfg'].get(
                            key,
                            default)))

    #
    # Compatibility with ConfigParser API
    #
    def sections(self):
        """Returns a list of the sections available"""
        return list(set([cfg_section_regex.match(k).group(1) for k in self._merged_store]))

    def options(self, section):
        """Returns a list of options available in the specified section."""
        opts = []
        for k in self._merged_store:
            sec, opt = cfg_sectionoption_regex.match(k).groups()
            if sec == section:
                opts.append(opt)
        return opts

    def has_section(self, section):
        """Indicates whether a section is present in the configuration"""
        for k in self._merged_store:
            if k.startswith(section):
                return True
        return False

    def has_option(self, section, option):
        """If the given section exists, and contains the given option"""
        for k in self._merged_store:
            sec, opt = cfg_sectionoption_regex.match(k).groups()
            if sec == section and opt == option:
                return True
        return False

    def _get_type(self, typefn, section, option):
        key = '.'.join([section, option])
        # Mimic the handling of get_value(..., default=None), while still going
        # through get() in order to get its default tuple handling.
        if key not in self:
            raise KeyError(key)
        return typefn(self.get(key))

    def getint(self, section, option):
        """A convenience method which coerces the option value to an integer"""
        return self._get_type(int, section, option)

    def getfloat(self, section, option):
        """A convenience method which coerces the option value to a float"""
        return self._get_type(float, section, option)

    def getbool(self, section, option, default=None):
        """A convenience method which coerces the option value to a bool

        Values "on", "yes", "true" and any int!=0 are considered True
        Values which evaluate to bool False, "off", "no", "false" are considered
        False
        TypeError is raised for other values.
        """
        key = '.'.join([section, option])
        # Mimic the handling of get_value(..., default=None), while still going
        # through get() in order to get its default tuple handling.
        if default is None and key not in self:
            raise KeyError(key)
        val = self.get(key, default=default)
        if val is None:  # no value at all, git treats it as True
            return True
        return anything2bool(val)

    # this is a hybrid of ConfigParser and dict API
    def items(self, section=None):
        """Return a list of (name, value) pairs for each option

        Optionally limited to a given section.
        """
        if section is None:
            return self._merged_store.items()
        return [(k, v) for k, v in self._merged_store.items()
                if cfg_section_regex.match(k).group(1) == section]

    #
    # Compatibility with GitPython's ConfigParser
    #
    def get_value(self, section, option, default=None):
        """Like `get()`, but with an optional default value

        If the default is not None, the given default value will be returned in
        case the option did not exist. This behavior imitates GitPython's
        config parser.
        """
        try:
            return self['.'.join((section, option))]
        except KeyError as e:
            # this strange dance is needed because gitpython does it this way
            if default is not None:
                return default
            else:
                raise e

    #
    # Modify configuration (proxy respective git-config call)
    #
    @_where_reload
    def _run(self, args, where=None, reload=False, **kwargs):
        """Centralized helper to run "git config" calls

        Parameters
        ----------
        args : list
          Arguments to pass for git config
        %s
        **kwargs
          Keywords arguments for Runner's call
        """
        if where:
            args = self._get_location_args(where) + args
        if '-l' in args:
            # we are just reading, no need to reload, no need to lock
            out = self._runner.run(self._config_cmd + args, **kwargs)
            return out['stdout'], out['stderr']

        # all other calls are modifications
        if '--file' in args:
            # all paths we are passing are absolute
            custom_file = Path(args[args.index('--file') + 1])
            custom_file.parent.mkdir(exist_ok=True)
        lockfile = None
        if self._repo_dot_git and ('--local' in args or '--file' in args):
            # modification of config in a dataset
            lockfile = self._repo_dot_git / 'config.dataladlock'
        else:
            # follow pattern in downloaders for lockfile location
            lockfile = Path(self.obtain('datalad.locations.cache')) \
                / 'locks' / 'gitconfig.lck'

        with ConfigManager._run_lock, InterProcessLock(lockfile, logger=lgr):
            out = self._runner.run(self._config_cmd + args, **kwargs)

        if reload:
            self.reload()
        return out['stdout'], out['stderr']

    def _get_location_args(self, where, args=None):
        if args is None:
            args = []
        cfg_labels = ('dataset', 'local', 'global', 'override')
        if where not in cfg_labels:
            raise ValueError(
                "unknown configuration label '{}' (not in {})".format(
                    where, cfg_labels))
        if where == 'dataset':
            if not self._repo_pathobj:
                raise ValueError(
                    'ConfigManager cannot store configuration to dataset, '
                    'none specified')
            dataset_cfgfile = self._repo_pathobj / DATASET_CONFIG_FILE
            args.extend(['--file', str(dataset_cfgfile)])
        elif where == 'global':
            args.append('--global')
        elif where == 'local':
            args.append('--local')
        return args

    @_where_reload
    def add(self, var, value, where='dataset', reload=True):
        """Add a configuration variable and value

        Parameters
        ----------
        var : str
          Variable name including any section like `git config` expects them, e.g.
          'core.editor'
        value : str
          Variable value
        %s"""
        if where == 'override':
            from datalad.utils import ensure_list
            val = ensure_list(self.overrides.pop(var, None))
            val.append(value)
            self.overrides[var] = val[0] if len(val) == 1 else val
            if reload:
                self.reload(force=True)
            return

        self._run(['--add', var, value], where=where, reload=reload,
                  protocol=StdOutErrCapture)

    @_where_reload
    def set(self, var, value, where='dataset', reload=True, force=False):
        """Set a variable to a value.

        In opposition to `add`, this replaces the value of `var` if there is
        one already.

        Parameters
        ----------
        var : str
          Variable name including any section like `git config` expects them, e.g.
          'core.editor'
        value : str
          Variable value
        force: bool
          if set, replaces all occurrences of `var` by a single one with the
          given `value`. Otherwise raise if multiple entries for `var` exist
          already
        %s"""
        if where == 'override':
            self.overrides[var] = value
            if reload:
                self.reload(force=True)
            return

        from datalad.support.gitrepo import to_options

        self._run(to_options(replace_all=force) + [var, value],
                  where=where, reload=reload, protocol=StdOutErrCapture)

    @_where_reload
    def rename_section(self, old, new, where='dataset', reload=True):
        """Rename a configuration section

        Parameters
        ----------
        old : str
          Name of the section to rename.
        new : str
          Name of the section to rename to.
        %s"""
        if where == 'override':
            self.overrides = {
                (new + k[len(old):]) if k.startswith(old + '.') else k: v
                for k, v in self.overrides.items()
            }
            if reload:
                self.reload(force=True)
            return

        self._run(['--rename-section', old, new], where=where, reload=reload)

    @_where_reload
    def remove_section(self, sec, where='dataset', reload=True):
        """Rename a configuration section

        Parameters
        ----------
        sec : str
          Name of the section to remove.
        %s"""
        if where == 'override':
            self.overrides = {
                k: v
                for k, v in self.overrides.items()
                if not k.startswith(sec + '.')
            }
            if reload:
                self.reload(force=True)
            return

        self._run(['--remove-section', sec], where=where, reload=reload)

    @_where_reload
    def unset(self, var, where='dataset', reload=True):
        """Remove all occurrences of a variable

        Parameters
        ----------
        var : str
          Name of the variable to remove
        %s"""
        if where == 'override':
            self.overrides.pop(var, None)
            if reload:
                self.reload(force=True)
            return

        # use unset all as it is simpler for now
        self._run(['--unset-all', var], where=where, reload=reload)


def rewrite_url(cfg, url):
    """Any matching 'url.<base>.insteadOf' configuration is applied

    Any URL that starts with such a configuration will be rewritten
    to start, instead, with <base>. When more than one insteadOf
    strings match a given URL, the longest match is used.

    Parameters
    ----------
    cfg : ConfigManager or dict
      dict-like with configuration variable name/value-pairs.
    url : str
      URL to be rewritten, if matching configuration is found.

    Returns
    -------
    str
      Rewritten or unmodified URL.
    """
    insteadof = {
        # only leave the base url
        k[4:-10]: v
        for k, v in cfg.items()
        if k.startswith('url.') and k.endswith('.insteadof')
    }

    # all config that applies
    matches = {
        key: v
        for key, val in insteadof.items()
        for v in (val if isinstance(val, tuple) else (val,))
        if url.startswith(v)
    }
    # find longest match, like Git does
    if matches:
        rewrite_base, match = sorted(
            matches.items(),
            key=lambda x: len(x[1]),
            reverse=True,
        )[0]
        if sum(match == v for v in matches.values()) > 1:
            lgr.warning(
                "Ignoring URL rewrite configuration for '%s', "
                "multiple conflicting definitions exists: %s",
                match,
                ['url.{}.insteadof'.format(k)
                 for k, v in matches.items()
                 if v == match]
            )
        else:
            url = '{}{}'.format(rewrite_base, url[len(match):])
    return url


# for convenience, bind to class too
ConfigManager.rewrite_url = rewrite_url

#
# Helpers for bypassing git-config when _writing_ config items,
# mostly useful when a large number of changes needs to be made
# and directly file manipulation without a safety net is worth
# the risk for performance reasons.
#

def quote_config(v):
    """Helper to perform minimal quoting of config keys/value parts

    Parameters
    ----------
    v : str
      To-be-quoted string
    """
    white = (' ', '\t')
    # backslashes need to be quoted in any case
    v = v.replace('\\', '\\\\')
    # must not have additional unquoted quotes
    v = v.replace('"', '\\"')
    if v[0] in white or v[-1] in white:
        # quoting the value due to leading/trailing whitespace
        v = '"{}"'.format(v)
    return v


def write_config_section(fobj, suite, name, props):
    """Write a config section with (multiple) settings.

    Parameters
    ----------
    fobj : File
       Opened target file
    suite : str
       First item of the section name, e.g. 'submodule', or
       'datalad'
    name : str
       Remainder of the section name
    props : dict
       Keys are configuration setting names within the section
       context (i.e. not duplicating `suite` and/or `name`, values
       are configuration setting values.
    """
    fmt = '[{_suite_} {_q_}{_name_}{_q_}]\n'
    for p in props:
        fmt += '\t{p} = {{{p}}}\n'.format(p=p)
    quoted_name = quote_config(name)
    fobj.write(
        fmt.format(
            _suite_=suite,
            _q_='' if quoted_name.startswith('"') else '"',
            _name_=quoted_name,
            **{k: quote_config(v) for k, v in props.items()}))

```
--- END FILE ---

Please first localize the bug based on the issue statement, and then generate *SEARCH/REPLACE* edits to fix the issue.

Every *SEARCH/REPLACE* edit must use this format:
1. The file path
2. The start of search block: <<<<<<< SEARCH
3. A contiguous chunk of lines to search for in the existing source code
4. The dividing line: =======
5. The lines to replace into the source code
6. The end of the replace block: >>>>>>> REPLACE

Here is an example:

```python
### mathweb/flask/app.py
<<<<<<< SEARCH
from flask import Flask
=======
import math
from flask import Flask
>>>>>>> REPLACE
```
</solution>

Please note that the *SEARCH/REPLACE* edit REQUIRES PROPER INDENTATION. If you would like to add the line '        print(x)', you must fully write that out, with all those spaces before the code!
Wrap each *SEARCH/REPLACE* edit in a code block as shown in the example above. If you have multiple *SEARCH/REPLACE* edits, use a separate code block for each one.
user