{
    "file_diffs": [
        {
            "old_file_content": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Interface information about credentials\n\nProvides minimalistic interface to deal (query, request, store) with most common\ntypes of credentials.  To be used by Authenticators\n\"\"\"\n\n__dev_doc__ = \"\"\"\nPossibly useful in the future 3rd part developments\n\nhttps://github.com/omab/python-social-auth\n    social authentication/registration mechanism with support for several\n    frameworks and auth providers\n\"\"\"\n\nimport time\n\nfrom collections import OrderedDict\n\nfrom ..dochelpers import exc_str\nfrom ..support.exceptions import AccessDeniedError\nfrom ..support.keyring_ import keyring as keyring_\nfrom ..ui import ui\nfrom ..utils import auto_repr\nfrom ..support.network import iso8601_to_epoch\n\nfrom datalad import cfg as dlcfg\n\nfrom logging import getLogger\nlgr = getLogger('datalad.downloaders.credentials')\n\n\n@auto_repr\nclass Credential(object):\n    \"\"\"Base class for different types of credentials\n    \"\"\"\n\n    # Should be defined in a subclass as an OrderedDict of fields\n    # name -> dict(attributes)\n    _FIELDS = None\n    _KNOWN_ATTRS = {\n        'hidden',    # UI should not display the value\n        'repeat',    # UI should repeat entry or not. Set to False to override default logic\n        'optional',  # Not mandatory thus not requested if not set\n    }\n\n    def __init__(self, name, url=None, keyring=None):\n        \"\"\"\n        Parameters\n        ----------\n        name : str\n            Name of the credential, as it would be identified by in the centralized\n            storage of credentials\n        url : str, optional\n            URL string to point users to where to seek obtaining the credentials\n        keyring : a keyring\n            An object providing (g|s)et_password.  If None, keyring module is used\n            as is\n        \"\"\"\n        self.name = name\n        self.url = url\n        self._keyring = keyring or keyring_\n        self._prepare()\n\n    def _prepare(self):\n        \"\"\"Additional house-keeping possibly to be performed in subclasses\n\n        Created to avoid all the passing args/kwargs of __init__ all the time\n        \"\"\"\n        # Basic checks\n        for f, fattrs in self._FIELDS.items():\n            unknown_attrs = set(fattrs).difference(self._KNOWN_ATTRS)\n            if unknown_attrs:\n                raise ValueError(\"Unknown attributes %s. Known are: %s\"\n                                 % (unknown_attrs, self._KNOWN_ATTRS))\n\n    def _is_field_optional(self, f):\n        return self._FIELDS[f].get('optional', False)\n\n    @property\n    def is_known(self):\n        \"\"\"Return True if values for all fields of the credential are known\"\"\"\n        try:\n            return all(\n                self._is_field_optional(f) or self._get_field_value(f) is not None\n                for f in self._FIELDS)\n        except Exception as exc:\n            lgr.warning(\"Failed to query keyring: %s\" % exc_str(exc))\n            return False\n\n    def _get_field_value(self, field):\n        return dlcfg.get('datalad.credential.{name}.{field}'.format(\n            name=self.name,\n            field=field.replace('_', '-')\n        )) or self._keyring.get(self.name, field)\n\n    def _ask_field_value(self, f, instructions=None):\n        msg = instructions if instructions else \\\n            (\"You need to authenticate with %r credentials.\" % self.name +\n                  (\" %s provides information on how to gain access\"\n                   % self.url if self.url else ''))\n\n        # provide custom options only if set for the field\n        f_props = self._FIELDS[f]\n        kwargs = {}\n        for p in ('hidden', 'repeat'):\n            if p in f_props:\n                kwargs[p] = f_props[p]\n        return ui.question(\n            f,\n            title=msg,\n            **kwargs\n        )\n\n    def _ask_and_set(self, f, instructions=None):\n        v = self._ask_field_value(f, instructions=instructions)\n        self.set(**{f: v})\n        return v\n\n    def enter_new(self, instructions=None, **kwargs):\n        \"\"\"Enter new values for the credential fields\n\n        Parameters\n        ----------\n        instructions : str, optional\n          If given, the auto-generated instructions based on a login-URL are\n          replaced by the given string\n        **kwargs\n          Any given key value pairs with non-None values are used to set the\n          field `key` to the given value, without asking for user input\n        \"\"\"\n        if kwargs:\n            unknown_fields = set(kwargs).difference(self._FIELDS)\n            known_fields = set(self._FIELDS).difference(kwargs)\n            if unknown_fields:\n                raise ValueError(\n                    \"Unknown to %s field(s): %s.  Known but not specified: %s\"\n                    % (self,\n                       ', '.join(sorted(unknown_fields)),\n                       ', '.join(sorted(known_fields))\n                       ))\n        # Use ui., request credential fields corresponding to the type\n        for f in self._FIELDS:\n            if kwargs.get(f, None):\n                # use given value, don't ask\n                self.set(**{f: kwargs[f]})\n            elif not self._is_field_optional(f):\n                self._ask_and_set(f, instructions=instructions)\n\n    def __call__(self):\n        \"\"\"Obtain credentials from a keyring and if any is not known -- ask\"\"\"\n        fields = {}\n        for f in self._FIELDS:\n            v = self._get_field_value(f)\n            if not self._is_field_optional(f):\n                while v is None:  # was not known\n                    v = self._ask_and_set(f)\n                fields[f] = v\n            elif v is not None:\n                fields[f] = v\n        return fields\n\n    def set(self, **kwargs):\n        \"\"\"Set field(s) of the credential\"\"\"\n        for f, v in kwargs.items():\n            if f not in self._FIELDS:\n                raise ValueError(\"Unknown field %s. Known are: %s\"\n                                 % (f, self._FIELDS.keys()))\n            self._keyring.set(self.name, f, v)\n\n    def get(self, f, default=None):\n        \"\"\"Get a field of the credential\"\"\"\n        if f not in self._FIELDS:\n            raise ValueError(\"Unknown field %s. Known are: %s\"\n                             % (f, self._FIELDS.keys()))\n        try:\n            return self._get_field_value(f)\n        except:  # MIH: what could even happen? _keyring not a dict?\n            return default\n\n    def delete(self):\n        \"\"\"Deletes credential values from the keyring\"\"\"\n        for f in self._FIELDS:\n            self._keyring.delete(self.name, f)\n\n\nclass UserPassword(Credential):\n    \"\"\"Simple type of a credential which consists of user/password pair\"\"\"\n\n    _FIELDS = OrderedDict([('user', {}),\n                           ('password', {'hidden': True})])\n\n    is_expired = False  # no expiration provisioned\n\n\nclass Token(Credential):\n    \"\"\"Simple type of a credential which provides a single token\"\"\"\n\n    _FIELDS = OrderedDict([('token', {'hidden': True, 'repeat': False})])\n\n    is_expired = False  # no expiration provisioned\n\n\nclass AWS_S3(Credential):\n    \"\"\"Credential for AWS S3 service\"\"\"\n\n    _FIELDS = OrderedDict([('key_id', {'repeat': False}),\n                           ('secret_id', {'hidden': True, 'repeat': False}),\n                           ('session', {'optional': True}),\n                           ('expiration', {'optional': True}),\n                           ])\n\n    @property\n    def is_expired(self):\n        exp = self.get('expiration', None)\n        if not exp:\n            return False\n        exp_epoch = iso8601_to_epoch(exp)\n        # -2 to artificially shorten duration of the allotment to avoid\n        # possible race conditions between us checking either it has\n        # already expired before submitting a request.\n        expire_in = (exp_epoch - time.time() - 2) / 3600.\n\n        lgr.debug(\n            (\"Credential %s has expired %.2fh ago\"\n                if expire_in <= 0 else \"Credential %s will expire in %.2fh\")\n            % (self.name, expire_in))\n        return expire_in <= 0\n\n\n@auto_repr\nclass CompositeCredential(Credential):\n    \"\"\"Credential which represent a sequence of Credentials where front one is exposed to user\n    \"\"\"\n\n    # To be defined in sub-classes\n    _CREDENTIAL_CLASSES = None\n    _CREDENTIAL_ADAPTERS = None\n\n    def _prepare(self):\n        assert len(self._CREDENTIAL_CLASSES) > 1, \"makes sense only if there is > 1 credential\"\n        assert len(self._CREDENTIAL_CLASSES) == len(self._CREDENTIAL_ADAPTERS) + 1, \\\n            \"there should be 1 less of adapter than _CREDENTIAL_CLASSES\"\n\n        for C in self._CREDENTIAL_CLASSES:\n            assert issubclass(C, Credential), \"%s must be a subclass of Credential\" % C\n\n        # First credential should bear the name and url\n        credentials = [self._CREDENTIAL_CLASSES[0](self.name, url=self.url, keyring=self._keyring)]\n        # and we just reuse its _FIELDS for _ask_field_value etc\n        self._FIELDS = credentials[0]._FIELDS\n        # the rest with index suffix, but storing themselves in the same keyring\n        for iC, C in enumerate(self._CREDENTIAL_CLASSES[1:]):\n            credentials.append(\n                C(name=\"%s:%d\" % (self.name, iC + 1), url=None, keyring=self._keyring)\n            )\n        self._credentials = credentials\n\n        super(CompositeCredential, self)._prepare()\n\n    # Here it becomes tricky, since theoretically it is the \"tail\"\n    # ones which might expire etc, so we wouldn't exactly know what\n    # new credentials outside process wanted -- it would be silly to ask\n    # right away the \"entry\" credentials if it is just the expiration of the\n    # tail credentials\n    def enter_new(self):\n        # should invalidate/remove all tail credentials to avoid failing attempts to login\n        self._credentials[0].enter_new()\n        self.refresh()\n\n    def refresh(self):\n        \"\"\"Re-establish \"dependent\" credentials\n\n        E.g. if code outside was reported that it expired somehow before known expiration datetime\n        \"\"\"\n        for c in self._credentials[1:]:\n            c.delete()\n        # trigger re-establishing the chain\n        _ = self()\n        if self.is_expired:\n            raise RuntimeError(\"Credential %s expired right upon refresh: should have not happened\")\n\n    @property\n    def is_expired(self):\n        return any(c.is_expired for c in self._credentials)\n\n    def __call__(self):\n        \"\"\"Obtain credentials from a keyring and if any is not known -- ask\"\"\"\n        # Start from the tail until we have credentials set\n        idx = len(self._credentials) - 1\n        for c in self._credentials[::-1]:\n            if c.is_known and not c.is_expired:\n                break\n            idx -= 1\n\n        if idx < 0:\n            # none was known, all the same -- start with the first one\n            idx = 0\n\n        # TODO: consider moving logic of traversal into authenticator since it is\n        # the one spitting out authentication error etc\n        # Theoretically we could just reuse 'fields' from adapter in the next step\n        # but let's do full circle, so that if any \"normalization\" is done by\n        # Credentials we take that into account\n        for c, adapter, next_c in zip(\n                self._credentials[idx:],\n                self._CREDENTIAL_ADAPTERS[idx:],\n                self._credentials[idx + 1:]):\n            fields = c()\n            next_fields = adapter(self, **fields)\n            next_c.set(**next_fields)\n\n        return self._credentials[-1]()\n\n\ndef _nda_adapter(composite, user=None, password=None):\n    from datalad.support.third.nda_aws_token_generator import NDATokenGenerator\n    from .. import cfg\n    nda_auth_url = cfg.obtain('datalad.externals.nda.dbserver')\n    gen = NDATokenGenerator(nda_auth_url)\n    lgr.debug(\"Generating token for NDA user %s using %s talking to %s\",\n              user, gen, nda_auth_url)\n    try:\n        token = gen.generate_token(user, password)\n    except Exception as exc:  # it is really just an \"Exception\"\n        exc_str = str(exc).lower()\n        # ATM it is \"Invalid username and/or password\"\n        # but who knows what future would bring\n        if \"invalid\" in exc_str and (\"user\" in exc_str or \"password\" in exc_str):\n            raise AccessDeniedError(exc_str)\n        raise\n    # There are also session and expiration we ignore... TODO anything about it?!!!\n    # we could create a derived AWS_S3 which would also store session and expiration\n    # and then may be Composite could use those????\n    return dict(key_id=token.access_key, secret_id=token.secret_key,\n                session=token.session, expiration=token.expiration)\n\n\nclass NDA_S3(CompositeCredential):\n    \"\"\"Credential to access NDA AWS\n\n    So for NDA we need a credential which is a composite credential.\n    User provides UserPassword and then some adapter generates AWS_S3\n    out of it\n    \"\"\"\n    _CREDENTIAL_CLASSES = (UserPassword, AWS_S3)\n    _CREDENTIAL_ADAPTERS = (_nda_adapter,)\n\n\ndef _loris_adapter(composite, user=None, password=None, **kwargs):\n    from datalad.support.third.loris_token_generator import LORISTokenGenerator\n\n    gen = LORISTokenGenerator(url=composite.url)\n    token = gen.generate_token(user, password)\n\n    return dict(token=token)\n\n\nclass LORIS_Token(CompositeCredential):\n    _CREDENTIAL_CLASSES = (UserPassword, Token)\n    _CREDENTIAL_ADAPTERS = (_loris_adapter,)\n\n    def __init__(self, name, url=None, keyring=None):\n        super(CompositeCredential, self).__init__(name, url, keyring)\n\n\nCREDENTIAL_TYPES = {\n    'user_password': UserPassword,\n    'aws-s3': AWS_S3,\n    'nda-s3': NDA_S3,\n    'token': Token,\n    'loris-token': LORIS_Token\n}\n",
            "new_file_content": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Interface information about credentials\n\nProvides minimalistic interface to deal (query, request, store) with most common\ntypes of credentials.  To be used by Authenticators\n\"\"\"\n\n__dev_doc__ = \"\"\"\nPossibly useful in the future 3rd part developments\n\nhttps://github.com/omab/python-social-auth\n    social authentication/registration mechanism with support for several\n    frameworks and auth providers\n\"\"\"\n\nimport time\n\nfrom collections import OrderedDict\n\nfrom ..dochelpers import exc_str\nfrom ..support.exceptions import AccessDeniedError\nfrom ..support.keyring_ import keyring as keyring_\nfrom ..ui import ui\nfrom ..utils import auto_repr\nfrom ..support.network import iso8601_to_epoch\n\nfrom datalad import cfg as dlcfg\nfrom datalad.config import anything2bool\n\nfrom logging import getLogger\nlgr = getLogger('datalad.downloaders.credentials')\n\n\n@auto_repr\nclass Credential(object):\n    \"\"\"Base class for different types of credentials\n    \"\"\"\n\n    # Should be defined in a subclass as an OrderedDict of fields\n    # name -> dict(attributes)\n    _FIELDS = None\n    _KNOWN_ATTRS = {\n        'hidden',    # UI should not display the value\n        'repeat',    # UI should repeat entry or not. Set to False to override default logic\n        'optional',  # Not mandatory thus not requested if not set\n    }\n\n    def __init__(self, name, url=None, keyring=None):\n        \"\"\"\n        Parameters\n        ----------\n        name : str\n            Name of the credential, as it would be identified by in the centralized\n            storage of credentials\n        url : str, optional\n            URL string to point users to where to seek obtaining the credentials\n        keyring : a keyring\n            An object providing (g|s)et_password.  If None, keyring module is used\n            as is\n        \"\"\"\n        self.name = name\n        self.url = url\n        self._keyring = keyring or keyring_\n        self._prepare()\n\n    def _prepare(self):\n        \"\"\"Additional house-keeping possibly to be performed in subclasses\n\n        Created to avoid all the passing args/kwargs of __init__ all the time\n        \"\"\"\n        # Basic checks\n        for f, fattrs in self._FIELDS.items():\n            unknown_attrs = set(fattrs).difference(self._KNOWN_ATTRS)\n            if unknown_attrs:\n                raise ValueError(\"Unknown attributes %s. Known are: %s\"\n                                 % (unknown_attrs, self._KNOWN_ATTRS))\n\n    def _is_field_optional(self, f):\n        return self._FIELDS[f].get('optional', False)\n\n    @property\n    def is_known(self):\n        \"\"\"Return True if values for all fields of the credential are known\"\"\"\n        try:\n            return all(\n                self._is_field_optional(f) or self._get_field_value(f) is not None\n                for f in self._FIELDS)\n        except Exception as exc:\n            lgr.warning(\"Failed to query keyring: %s\" % exc_str(exc))\n            return False\n\n    def _get_field_value(self, field):\n        return dlcfg.get('datalad.credential.{name}.{field}'.format(\n            name=self.name,\n            field=field.replace('_', '-')\n        )) or self._keyring.get(self.name, field)\n\n    def _ask_field_value(self, f, instructions=None):\n        msg = instructions if instructions else \\\n            (\"You need to authenticate with %r credentials.\" % self.name +\n                  (\" %s provides information on how to gain access\"\n                   % self.url if self.url else ''))\n\n        # provide custom options only if set for the field\n        f_props = self._FIELDS[f]\n        kwargs = {}\n        for p in ('hidden', 'repeat'):\n            if p in f_props:\n                kwargs[p] = f_props[p]\n        return ui.question(\n            f,\n            title=msg,\n            **kwargs\n        )\n\n    def _ask_and_set(self, f, instructions=None):\n        v = self._ask_field_value(f, instructions=instructions)\n        self.set(**{f: v})\n        return v\n\n    def enter_new(self, instructions=None, **kwargs):\n        \"\"\"Enter new values for the credential fields\n\n        Parameters\n        ----------\n        instructions : str, optional\n          If given, the auto-generated instructions based on a login-URL are\n          replaced by the given string\n        **kwargs\n          Any given key value pairs with non-None values are used to set the\n          field `key` to the given value, without asking for user input\n        \"\"\"\n        if kwargs:\n            unknown_fields = set(kwargs).difference(self._FIELDS)\n            known_fields = set(self._FIELDS).difference(kwargs)\n            if unknown_fields:\n                raise ValueError(\n                    \"Unknown to %s field(s): %s.  Known but not specified: %s\"\n                    % (self,\n                       ', '.join(sorted(unknown_fields)),\n                       ', '.join(sorted(known_fields))\n                       ))\n        # Use ui., request credential fields corresponding to the type\n        for f in self._FIELDS:\n            if kwargs.get(f, None):\n                # use given value, don't ask\n                self.set(**{f: kwargs[f]})\n            elif not self._is_field_optional(f):\n                self._ask_and_set(f, instructions=instructions)\n\n    def __call__(self):\n        \"\"\"Obtain credentials from a keyring and if any is not known -- ask\"\"\"\n        fields = {}\n        # check if we shall ask for credentials, even if some are on record\n        # already (but maybe they were found to need updating)\n        force_reentry = dlcfg.obtain(\n            'datalad.credentials.force-ask',\n            valtype=anything2bool)\n        for f in self._FIELDS:\n            # don't query for value if we need to get a new one\n            v = None if force_reentry else self._get_field_value(f)\n            if not self._is_field_optional(f):\n                while v is None:  # was not known\n                    v = self._ask_and_set(f)\n                fields[f] = v\n            elif v is not None:\n                fields[f] = v\n        return fields\n\n    def set(self, **kwargs):\n        \"\"\"Set field(s) of the credential\"\"\"\n        for f, v in kwargs.items():\n            if f not in self._FIELDS:\n                raise ValueError(\"Unknown field %s. Known are: %s\"\n                                 % (f, self._FIELDS.keys()))\n            self._keyring.set(self.name, f, v)\n\n    def get(self, f, default=None):\n        \"\"\"Get a field of the credential\"\"\"\n        if f not in self._FIELDS:\n            raise ValueError(\"Unknown field %s. Known are: %s\"\n                             % (f, self._FIELDS.keys()))\n        try:\n            return self._get_field_value(f)\n        except:  # MIH: what could even happen? _keyring not a dict?\n            return default\n\n    def delete(self):\n        \"\"\"Deletes credential values from the keyring\"\"\"\n        for f in self._FIELDS:\n            self._keyring.delete(self.name, f)\n\n\nclass UserPassword(Credential):\n    \"\"\"Simple type of a credential which consists of user/password pair\"\"\"\n\n    _FIELDS = OrderedDict([('user', {}),\n                           ('password', {'hidden': True})])\n\n    is_expired = False  # no expiration provisioned\n\n\nclass Token(Credential):\n    \"\"\"Simple type of a credential which provides a single token\"\"\"\n\n    _FIELDS = OrderedDict([('token', {'hidden': True, 'repeat': False})])\n\n    is_expired = False  # no expiration provisioned\n\n\nclass AWS_S3(Credential):\n    \"\"\"Credential for AWS S3 service\"\"\"\n\n    _FIELDS = OrderedDict([('key_id', {'repeat': False}),\n                           ('secret_id', {'hidden': True, 'repeat': False}),\n                           ('session', {'optional': True}),\n                           ('expiration', {'optional': True}),\n                           ])\n\n    @property\n    def is_expired(self):\n        exp = self.get('expiration', None)\n        if not exp:\n            return False\n        exp_epoch = iso8601_to_epoch(exp)\n        # -2 to artificially shorten duration of the allotment to avoid\n        # possible race conditions between us checking either it has\n        # already expired before submitting a request.\n        expire_in = (exp_epoch - time.time() - 2) / 3600.\n\n        lgr.debug(\n            (\"Credential %s has expired %.2fh ago\"\n                if expire_in <= 0 else \"Credential %s will expire in %.2fh\")\n            % (self.name, expire_in))\n        return expire_in <= 0\n\n\n@auto_repr\nclass CompositeCredential(Credential):\n    \"\"\"Credential which represent a sequence of Credentials where front one is exposed to user\n    \"\"\"\n\n    # To be defined in sub-classes\n    _CREDENTIAL_CLASSES = None\n    _CREDENTIAL_ADAPTERS = None\n\n    def _prepare(self):\n        assert len(self._CREDENTIAL_CLASSES) > 1, \"makes sense only if there is > 1 credential\"\n        assert len(self._CREDENTIAL_CLASSES) == len(self._CREDENTIAL_ADAPTERS) + 1, \\\n            \"there should be 1 less of adapter than _CREDENTIAL_CLASSES\"\n\n        for C in self._CREDENTIAL_CLASSES:\n            assert issubclass(C, Credential), \"%s must be a subclass of Credential\" % C\n\n        # First credential should bear the name and url\n        credentials = [self._CREDENTIAL_CLASSES[0](self.name, url=self.url, keyring=self._keyring)]\n        # and we just reuse its _FIELDS for _ask_field_value etc\n        self._FIELDS = credentials[0]._FIELDS\n        # the rest with index suffix, but storing themselves in the same keyring\n        for iC, C in enumerate(self._CREDENTIAL_CLASSES[1:]):\n            credentials.append(\n                C(name=\"%s:%d\" % (self.name, iC + 1), url=None, keyring=self._keyring)\n            )\n        self._credentials = credentials\n\n        super(CompositeCredential, self)._prepare()\n\n    # Here it becomes tricky, since theoretically it is the \"tail\"\n    # ones which might expire etc, so we wouldn't exactly know what\n    # new credentials outside process wanted -- it would be silly to ask\n    # right away the \"entry\" credentials if it is just the expiration of the\n    # tail credentials\n    def enter_new(self):\n        # should invalidate/remove all tail credentials to avoid failing attempts to login\n        self._credentials[0].enter_new()\n        self.refresh()\n\n    def refresh(self):\n        \"\"\"Re-establish \"dependent\" credentials\n\n        E.g. if code outside was reported that it expired somehow before known expiration datetime\n        \"\"\"\n        for c in self._credentials[1:]:\n            c.delete()\n        # trigger re-establishing the chain\n        _ = self()\n        if self.is_expired:\n            raise RuntimeError(\"Credential %s expired right upon refresh: should have not happened\")\n\n    @property\n    def is_expired(self):\n        return any(c.is_expired for c in self._credentials)\n\n    def __call__(self):\n        \"\"\"Obtain credentials from a keyring and if any is not known -- ask\"\"\"\n        # Start from the tail until we have credentials set\n        idx = len(self._credentials) - 1\n        for c in self._credentials[::-1]:\n            if c.is_known and not c.is_expired:\n                break\n            idx -= 1\n\n        if idx < 0:\n            # none was known, all the same -- start with the first one\n            idx = 0\n\n        # TODO: consider moving logic of traversal into authenticator since it is\n        # the one spitting out authentication error etc\n        # Theoretically we could just reuse 'fields' from adapter in the next step\n        # but let's do full circle, so that if any \"normalization\" is done by\n        # Credentials we take that into account\n        for c, adapter, next_c in zip(\n                self._credentials[idx:],\n                self._CREDENTIAL_ADAPTERS[idx:],\n                self._credentials[idx + 1:]):\n            fields = c()\n            next_fields = adapter(self, **fields)\n            next_c.set(**next_fields)\n\n        return self._credentials[-1]()\n\n\ndef _nda_adapter(composite, user=None, password=None):\n    from datalad.support.third.nda_aws_token_generator import NDATokenGenerator\n    from .. import cfg\n    nda_auth_url = cfg.obtain('datalad.externals.nda.dbserver')\n    gen = NDATokenGenerator(nda_auth_url)\n    lgr.debug(\"Generating token for NDA user %s using %s talking to %s\",\n              user, gen, nda_auth_url)\n    try:\n        token = gen.generate_token(user, password)\n    except Exception as exc:  # it is really just an \"Exception\"\n        exc_str = str(exc).lower()\n        # ATM it is \"Invalid username and/or password\"\n        # but who knows what future would bring\n        if \"invalid\" in exc_str and (\"user\" in exc_str or \"password\" in exc_str):\n            raise AccessDeniedError(exc_str)\n        raise\n    # There are also session and expiration we ignore... TODO anything about it?!!!\n    # we could create a derived AWS_S3 which would also store session and expiration\n    # and then may be Composite could use those????\n    return dict(key_id=token.access_key, secret_id=token.secret_key,\n                session=token.session, expiration=token.expiration)\n\n\nclass NDA_S3(CompositeCredential):\n    \"\"\"Credential to access NDA AWS\n\n    So for NDA we need a credential which is a composite credential.\n    User provides UserPassword and then some adapter generates AWS_S3\n    out of it\n    \"\"\"\n    _CREDENTIAL_CLASSES = (UserPassword, AWS_S3)\n    _CREDENTIAL_ADAPTERS = (_nda_adapter,)\n\n\ndef _loris_adapter(composite, user=None, password=None, **kwargs):\n    from datalad.support.third.loris_token_generator import LORISTokenGenerator\n\n    gen = LORISTokenGenerator(url=composite.url)\n    token = gen.generate_token(user, password)\n\n    return dict(token=token)\n\n\nclass LORIS_Token(CompositeCredential):\n    _CREDENTIAL_CLASSES = (UserPassword, Token)\n    _CREDENTIAL_ADAPTERS = (_loris_adapter,)\n\n    def __init__(self, name, url=None, keyring=None):\n        super(CompositeCredential, self).__init__(name, url, keyring)\n\n\nCREDENTIAL_TYPES = {\n    'user_password': UserPassword,\n    'aws-s3': AWS_S3,\n    'nda-s3': NDA_S3,\n    'token': Token,\n    'loris-token': LORIS_Token\n}\n",
            "header": {
                "file": {
                    "path": "datalad/downloaders/credentials.py"
                },
                "misc_line": null
            },
            "index_line": {
                "old_commit_hash": "063038936",
                "new_commit_hash": "21aaea39a",
                "mode": "100644"
            },
            "is_binary_file": false,
            "binary_line": null,
            "minus_file": {
                "path": "a/datalad/downloaders/credentials.py"
            },
            "plus_file": {
                "path": "b/datalad/downloaders/credentials.py"
            },
            "hunks": [
                {
                    "descriptor": {
                        "old_range": {
                            "start": 32,
                            "length": 6
                        },
                        "new_range": {
                            "start": 32,
                            "length": 7
                        },
                        "section": "from ..utils import auto_repr"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "from ..support.network import iso8601_to_epoch",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "from datalad import cfg as dlcfg",
                                "type": "context"
                            },
                            {
                                "content": "from datalad.config import anything2bool",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "from logging import getLogger",
                                "type": "context"
                            },
                            {
                                "content": "lgr = getLogger('datalad.downloaders.credentials')",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [],
                    "added_entities": [
                        {
                            "file_name": "datalad/downloaders/credentials.py",
                            "type": "statement",
                            "name": "",
                            "content": "from datalad.config import anything2bool",
                            "ast_type_str": "ImportFrom",
                            "start_lineno": 35,
                            "end_lineno": 35,
                            "parent": null
                        }
                    ],
                    "deleted_entities": []
                },
                {
                    "descriptor": {
                        "old_range": {
                            "start": 157,
                            "length": 8
                        },
                        "new_range": {
                            "start": 158,
                            "length": 14
                        },
                        "section": "class Credential(object):"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "    def __call__(self):",
                                "type": "context"
                            },
                            {
                                "content": "        \"\"\"Obtain credentials from a keyring and if any is not known -- ask\"\"\"",
                                "type": "context"
                            },
                            {
                                "content": "        fields = {}",
                                "type": "context"
                            },
                            {
                                "content": "        # check if we shall ask for credentials, even if some are on record",
                                "type": "added"
                            },
                            {
                                "content": "        # already (but maybe they were found to need updating)",
                                "type": "added"
                            },
                            {
                                "content": "        force_reentry = dlcfg.obtain(",
                                "type": "added"
                            },
                            {
                                "content": "            'datalad.credentials.force-ask',",
                                "type": "added"
                            },
                            {
                                "content": "            valtype=anything2bool)",
                                "type": "added"
                            },
                            {
                                "content": "        for f in self._FIELDS:",
                                "type": "context"
                            },
                            {
                                "content": "            v = self._get_field_value(f)",
                                "type": "deleted"
                            },
                            {
                                "content": "            # don't query for value if we need to get a new one",
                                "type": "added"
                            },
                            {
                                "content": "            v = None if force_reentry else self._get_field_value(f)",
                                "type": "added"
                            },
                            {
                                "content": "            if not self._is_field_optional(f):",
                                "type": "context"
                            },
                            {
                                "content": "                while v is None:  # was not known",
                                "type": "context"
                            },
                            {
                                "content": "                    v = self._ask_and_set(f)",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [
                        {
                            "file_name": "datalad/downloaders/credentials.py",
                            "type": "method",
                            "name": "Credential.__call__",
                            "content": "class Credential(object):\n    \"\"\"Base class for different types of credentials\n    \"\"\"\n\n    # Should be defined in a subclass as an OrderedDict of fields\n    # name -> dict(attributes)\n    _FIELDS = None\n    _KNOWN_ATTRS = {\n        'hidden',    # UI should not display the value\n        'repeat',    # UI should repeat entry or not. Set to False to override default logic\n        'optional',  # Not mandatory thus not requested if not set\n    }\n\n    def __init__(self, name, url=None, keyring=None):\n        \"\"\"\n        Parameters\n        ----------\n        name : str\n            Name of the credential, as it would be identified by in the centralized\n            storage of credentials\n        url : str, optional\n            URL string to point users to where to seek obtaining the credentials\n        keyring : a keyring\n            An object providing (g|s)et_password.  If None, keyring module is used\n            as is\n        \"\"\"\n        self.name = name\n        self.url = url\n        self._keyring = keyring or keyring_\n        self._prepare()\n\n    def _prepare(self):\n        \"\"\"Additional house-keeping possibly to be performed in subclasses\n\n        Created to avoid all the passing args/kwargs of __init__ all the time\n        \"\"\"\n        # Basic checks\n        for f, fattrs in self._FIELDS.items():\n            unknown_attrs = set(fattrs).difference(self._KNOWN_ATTRS)\n            if unknown_attrs:\n                raise ValueError(\"Unknown attributes %s. Known are: %s\"\n                                 % (unknown_attrs, self._KNOWN_ATTRS))\n\n    def _is_field_optional(self, f):\n        return self._FIELDS[f].get('optional', False)\n\n    @property\n    def is_known(self):\n        \"\"\"Return True if values for all fields of the credential are known\"\"\"\n        try:\n            return all(\n                self._is_field_optional(f) or self._get_field_value(f) is not None\n                for f in self._FIELDS)\n        except Exception as exc:\n            lgr.warning(\"Failed to query keyring: %s\" % exc_str(exc))\n            return False\n\n    def _get_field_value(self, field):\n        return dlcfg.get('datalad.credential.{name}.{field}'.format(\n            name=self.name,\n            field=field.replace('_', '-')\n        )) or self._keyring.get(self.name, field)\n\n    def _ask_field_value(self, f, instructions=None):\n        msg = instructions if instructions else \\\n            (\"You need to authenticate with %r credentials.\" % self.name +\n                  (\" %s provides information on how to gain access\"\n                   % self.url if self.url else ''))\n\n        # provide custom options only if set for the field\n        f_props = self._FIELDS[f]\n        kwargs = {}\n        for p in ('hidden', 'repeat'):\n            if p in f_props:\n                kwargs[p] = f_props[p]\n        return ui.question(\n            f,\n            title=msg,\n            **kwargs\n        )\n\n    def _ask_and_set(self, f, instructions=None):\n        v = self._ask_field_value(f, instructions=instructions)\n        self.set(**{f: v})\n        return v\n\n    def enter_new(self, instructions=None, **kwargs):\n        \"\"\"Enter new values for the credential fields\n\n        Parameters\n        ----------\n        instructions : str, optional\n          If given, the auto-generated instructions based on a login-URL are\n          replaced by the given string\n        **kwargs\n          Any given key value pairs with non-None values are used to set the\n          field `key` to the given value, without asking for user input\n        \"\"\"\n        if kwargs:\n            unknown_fields = set(kwargs).difference(self._FIELDS)\n            known_fields = set(self._FIELDS).difference(kwargs)\n            if unknown_fields:\n                raise ValueError(\n                    \"Unknown to %s field(s): %s.  Known but not specified: %s\"\n                    % (self,\n                       ', '.join(sorted(unknown_fields)),\n                       ', '.join(sorted(known_fields))\n                       ))\n        # Use ui., request credential fields corresponding to the type\n        for f in self._FIELDS:\n            if kwargs.get(f, None):\n                # use given value, don't ask\n                self.set(**{f: kwargs[f]})\n            elif not self._is_field_optional(f):\n                self._ask_and_set(f, instructions=instructions)\n\n    def __call__(self):\n        \"\"\"Obtain credentials from a keyring and if any is not known -- ask\"\"\"\n        fields = {}\n        for f in self._FIELDS:\n            v = self._get_field_value(f)\n            if not self._is_field_optional(f):\n                while v is None:  # was not known\n                    v = self._ask_and_set(f)\n                fields[f] = v\n            elif v is not None:\n                fields[f] = v\n        return fields\n\n    def set(self, **kwargs):\n        \"\"\"Set field(s) of the credential\"\"\"\n        for f, v in kwargs.items():\n            if f not in self._FIELDS:\n                raise ValueError(\"Unknown field %s. Known are: %s\"\n                                 % (f, self._FIELDS.keys()))\n            self._keyring.set(self.name, f, v)\n\n    def get(self, f, default=None):\n        \"\"\"Get a field of the credential\"\"\"\n        if f not in self._FIELDS:\n            raise ValueError(\"Unknown field %s. Known are: %s\"\n                             % (f, self._FIELDS.keys()))\n        try:\n            return self._get_field_value(f)\n        except:  # MIH: what could even happen? _keyring not a dict?\n            return default\n\n    def delete(self):\n        \"\"\"Deletes credential values from the keyring\"\"\"\n        for f in self._FIELDS:\n            self._keyring.delete(self.name, f)",
                            "ast_type_str": "ClassDef",
                            "start_lineno": 157,
                            "end_lineno": 168,
                            "parent": {
                                "file_name": "datalad/downloaders/credentials.py",
                                "type": "class",
                                "name": "Credential",
                                "content": "class Credential(object):\n    \"\"\"Base class for different types of credentials\n    \"\"\"\n\n    # Should be defined in a subclass as an OrderedDict of fields\n    # name -> dict(attributes)\n    _FIELDS = None\n    _KNOWN_ATTRS = {\n        'hidden',    # UI should not display the value\n        'repeat',    # UI should repeat entry or not. Set to False to override default logic\n        'optional',  # Not mandatory thus not requested if not set\n    }\n\n    def __init__(self, name, url=None, keyring=None):\n        \"\"\"\n        Parameters\n        ----------\n        name : str\n            Name of the credential, as it would be identified by in the centralized\n            storage of credentials\n        url : str, optional\n            URL string to point users to where to seek obtaining the credentials\n        keyring : a keyring\n            An object providing (g|s)et_password.  If None, keyring module is used\n            as is\n        \"\"\"\n        self.name = name\n        self.url = url\n        self._keyring = keyring or keyring_\n        self._prepare()\n\n    def _prepare(self):\n        \"\"\"Additional house-keeping possibly to be performed in subclasses\n\n        Created to avoid all the passing args/kwargs of __init__ all the time\n        \"\"\"\n        # Basic checks\n        for f, fattrs in self._FIELDS.items():\n            unknown_attrs = set(fattrs).difference(self._KNOWN_ATTRS)\n            if unknown_attrs:\n                raise ValueError(\"Unknown attributes %s. Known are: %s\"\n                                 % (unknown_attrs, self._KNOWN_ATTRS))\n\n    def _is_field_optional(self, f):\n        return self._FIELDS[f].get('optional', False)\n\n    @property\n    def is_known(self):\n        \"\"\"Return True if values for all fields of the credential are known\"\"\"\n        try:\n            return all(\n                self._is_field_optional(f) or self._get_field_value(f) is not None\n                for f in self._FIELDS)\n        except Exception as exc:\n            lgr.warning(\"Failed to query keyring: %s\" % exc_str(exc))\n            return False\n\n    def _get_field_value(self, field):\n        return dlcfg.get('datalad.credential.{name}.{field}'.format(\n            name=self.name,\n            field=field.replace('_', '-')\n        )) or self._keyring.get(self.name, field)\n\n    def _ask_field_value(self, f, instructions=None):\n        msg = instructions if instructions else \\\n            (\"You need to authenticate with %r credentials.\" % self.name +\n                  (\" %s provides information on how to gain access\"\n                   % self.url if self.url else ''))\n\n        # provide custom options only if set for the field\n        f_props = self._FIELDS[f]\n        kwargs = {}\n        for p in ('hidden', 'repeat'):\n            if p in f_props:\n                kwargs[p] = f_props[p]\n        return ui.question(\n            f,\n            title=msg,\n            **kwargs\n        )\n\n    def _ask_and_set(self, f, instructions=None):\n        v = self._ask_field_value(f, instructions=instructions)\n        self.set(**{f: v})\n        return v\n\n    def enter_new(self, instructions=None, **kwargs):\n        \"\"\"Enter new values for the credential fields\n\n        Parameters\n        ----------\n        instructions : str, optional\n          If given, the auto-generated instructions based on a login-URL are\n          replaced by the given string\n        **kwargs\n          Any given key value pairs with non-None values are used to set the\n          field `key` to the given value, without asking for user input\n        \"\"\"\n        if kwargs:\n            unknown_fields = set(kwargs).difference(self._FIELDS)\n            known_fields = set(self._FIELDS).difference(kwargs)\n            if unknown_fields:\n                raise ValueError(\n                    \"Unknown to %s field(s): %s.  Known but not specified: %s\"\n                    % (self,\n                       ', '.join(sorted(unknown_fields)),\n                       ', '.join(sorted(known_fields))\n                       ))\n        # Use ui., request credential fields corresponding to the type\n        for f in self._FIELDS:\n            if kwargs.get(f, None):\n                # use given value, don't ask\n                self.set(**{f: kwargs[f]})\n            elif not self._is_field_optional(f):\n                self._ask_and_set(f, instructions=instructions)\n\n    def __call__(self):\n        \"\"\"Obtain credentials from a keyring and if any is not known -- ask\"\"\"\n        fields = {}\n        for f in self._FIELDS:\n            v = self._get_field_value(f)\n            if not self._is_field_optional(f):\n                while v is None:  # was not known\n                    v = self._ask_and_set(f)\n                fields[f] = v\n            elif v is not None:\n                fields[f] = v\n        return fields\n\n    def set(self, **kwargs):\n        \"\"\"Set field(s) of the credential\"\"\"\n        for f, v in kwargs.items():\n            if f not in self._FIELDS:\n                raise ValueError(\"Unknown field %s. Known are: %s\"\n                                 % (f, self._FIELDS.keys()))\n            self._keyring.set(self.name, f, v)\n\n    def get(self, f, default=None):\n        \"\"\"Get a field of the credential\"\"\"\n        if f not in self._FIELDS:\n            raise ValueError(\"Unknown field %s. Known are: %s\"\n                             % (f, self._FIELDS.keys()))\n        try:\n            return self._get_field_value(f)\n        except:  # MIH: what could even happen? _keyring not a dict?\n            return default\n\n    def delete(self):\n        \"\"\"Deletes credential values from the keyring\"\"\"\n        for f in self._FIELDS:\n            self._keyring.delete(self.name, f)",
                                "ast_type_str": "ClassDef",
                                "start_lineno": 41,
                                "end_lineno": 191,
                                "parent": null
                            }
                        },
                        {
                            "file_name": "datalad/downloaders/credentials.py",
                            "type": "class",
                            "name": "Credential",
                            "content": "class Credential(object):\n    \"\"\"Base class for different types of credentials\n    \"\"\"\n\n    # Should be defined in a subclass as an OrderedDict of fields\n    # name -> dict(attributes)\n    _FIELDS = None\n    _KNOWN_ATTRS = {\n        'hidden',    # UI should not display the value\n        'repeat',    # UI should repeat entry or not. Set to False to override default logic\n        'optional',  # Not mandatory thus not requested if not set\n    }\n\n    def __init__(self, name, url=None, keyring=None):\n        \"\"\"\n        Parameters\n        ----------\n        name : str\n            Name of the credential, as it would be identified by in the centralized\n            storage of credentials\n        url : str, optional\n            URL string to point users to where to seek obtaining the credentials\n        keyring : a keyring\n            An object providing (g|s)et_password.  If None, keyring module is used\n            as is\n        \"\"\"\n        self.name = name\n        self.url = url\n        self._keyring = keyring or keyring_\n        self._prepare()\n\n    def _prepare(self):\n        \"\"\"Additional house-keeping possibly to be performed in subclasses\n\n        Created to avoid all the passing args/kwargs of __init__ all the time\n        \"\"\"\n        # Basic checks\n        for f, fattrs in self._FIELDS.items():\n            unknown_attrs = set(fattrs).difference(self._KNOWN_ATTRS)\n            if unknown_attrs:\n                raise ValueError(\"Unknown attributes %s. Known are: %s\"\n                                 % (unknown_attrs, self._KNOWN_ATTRS))\n\n    def _is_field_optional(self, f):\n        return self._FIELDS[f].get('optional', False)\n\n    @property\n    def is_known(self):\n        \"\"\"Return True if values for all fields of the credential are known\"\"\"\n        try:\n            return all(\n                self._is_field_optional(f) or self._get_field_value(f) is not None\n                for f in self._FIELDS)\n        except Exception as exc:\n            lgr.warning(\"Failed to query keyring: %s\" % exc_str(exc))\n            return False\n\n    def _get_field_value(self, field):\n        return dlcfg.get('datalad.credential.{name}.{field}'.format(\n            name=self.name,\n            field=field.replace('_', '-')\n        )) or self._keyring.get(self.name, field)\n\n    def _ask_field_value(self, f, instructions=None):\n        msg = instructions if instructions else \\\n            (\"You need to authenticate with %r credentials.\" % self.name +\n                  (\" %s provides information on how to gain access\"\n                   % self.url if self.url else ''))\n\n        # provide custom options only if set for the field\n        f_props = self._FIELDS[f]\n        kwargs = {}\n        for p in ('hidden', 'repeat'):\n            if p in f_props:\n                kwargs[p] = f_props[p]\n        return ui.question(\n            f,\n            title=msg,\n            **kwargs\n        )\n\n    def _ask_and_set(self, f, instructions=None):\n        v = self._ask_field_value(f, instructions=instructions)\n        self.set(**{f: v})\n        return v\n\n    def enter_new(self, instructions=None, **kwargs):\n        \"\"\"Enter new values for the credential fields\n\n        Parameters\n        ----------\n        instructions : str, optional\n          If given, the auto-generated instructions based on a login-URL are\n          replaced by the given string\n        **kwargs\n          Any given key value pairs with non-None values are used to set the\n          field `key` to the given value, without asking for user input\n        \"\"\"\n        if kwargs:\n            unknown_fields = set(kwargs).difference(self._FIELDS)\n            known_fields = set(self._FIELDS).difference(kwargs)\n            if unknown_fields:\n                raise ValueError(\n                    \"Unknown to %s field(s): %s.  Known but not specified: %s\"\n                    % (self,\n                       ', '.join(sorted(unknown_fields)),\n                       ', '.join(sorted(known_fields))\n                       ))\n        # Use ui., request credential fields corresponding to the type\n        for f in self._FIELDS:\n            if kwargs.get(f, None):\n                # use given value, don't ask\n                self.set(**{f: kwargs[f]})\n            elif not self._is_field_optional(f):\n                self._ask_and_set(f, instructions=instructions)\n\n    def __call__(self):\n        \"\"\"Obtain credentials from a keyring and if any is not known -- ask\"\"\"\n        fields = {}\n        for f in self._FIELDS:\n            v = self._get_field_value(f)\n            if not self._is_field_optional(f):\n                while v is None:  # was not known\n                    v = self._ask_and_set(f)\n                fields[f] = v\n            elif v is not None:\n                fields[f] = v\n        return fields\n\n    def set(self, **kwargs):\n        \"\"\"Set field(s) of the credential\"\"\"\n        for f, v in kwargs.items():\n            if f not in self._FIELDS:\n                raise ValueError(\"Unknown field %s. Known are: %s\"\n                                 % (f, self._FIELDS.keys()))\n            self._keyring.set(self.name, f, v)\n\n    def get(self, f, default=None):\n        \"\"\"Get a field of the credential\"\"\"\n        if f not in self._FIELDS:\n            raise ValueError(\"Unknown field %s. Known are: %s\"\n                             % (f, self._FIELDS.keys()))\n        try:\n            return self._get_field_value(f)\n        except:  # MIH: what could even happen? _keyring not a dict?\n            return default\n\n    def delete(self):\n        \"\"\"Deletes credential values from the keyring\"\"\"\n        for f in self._FIELDS:\n            self._keyring.delete(self.name, f)",
                            "ast_type_str": "ClassDef",
                            "start_lineno": 41,
                            "end_lineno": 191,
                            "parent": null
                        }
                    ],
                    "added_entities": [],
                    "deleted_entities": []
                }
            ]
        },
        {
            "old_file_content": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Tests for credentials\"\"\"\n\nfrom unittest.mock import patch\nfrom datalad.tests.utils import (\n    assert_equal,\n    assert_false,\n    assert_in,\n    assert_raises,\n    assert_true,\n    SkipTest,\n    with_testsui,\n)\nfrom datalad.support.keyring_ import (\n    Keyring,\n    MemoryKeyring,\n)\nfrom ..credentials import (\n    AWS_S3,\n    CompositeCredential,\n    UserPassword,\n)\nfrom datalad.config import ConfigManager\n\n\n@with_testsui(responses=[\n    'user1', 'password1',\n    # when we do provide user to enter_new\n    'newpassword',\n])\ndef test_cred1_enter_new():\n    keyring = MemoryKeyring()\n    cred = UserPassword(\"name\", keyring=keyring)\n    assert_false(cred.is_known)\n    assert_equal(cred.enter_new(), None)\n    assert_true(cred.is_known)\n    assert_equal(keyring.get('name', 'user'), 'user1')\n    assert_equal(keyring.get('name', 'password'), 'password1')\n    keyring.delete('name')\n    assert_raises(KeyError, keyring.delete, 'name', 'user')\n    assert_raises(KeyError, keyring.delete, 'name')\n    assert_equal(keyring.get('name', 'user'), None)\n\n    # Test it blowing up if we provide unknown field\n    with assert_raises(ValueError) as cme:\n        cred.enter_new(username='user')\n    assert_in('field(s): username.  Known but not specified: password, user',\n              str(cme.exception))\n\n    # Test that if user is provided, it is not asked\n    cred.enter_new(user='user2')\n    assert_equal(keyring.get('name', 'user'), 'user2')\n    assert_equal(keyring.get('name', 'password'), 'newpassword')\n\n@with_testsui(responses=['password1'])\ndef test_cred1_call():\n    keyring = MemoryKeyring()\n    cred = UserPassword(\"name\", keyring=keyring)\n    # we will set the name but not the password, expecting UI\n    # requesting it\n    assert_equal(keyring.set('name', 'user', 'user1'), None)\n    assert_equal(keyring.get('name', 'user'), 'user1')\n    assert_equal(cred(), {'user': 'user1', 'password': 'password1'})\n    assert_equal(keyring.get('name', 'password'), 'password1')\n\n\ndef test_keyring():\n    # mock out keyring methods and test that we are providing correct values\n    # with 'datalad-' prefix\n    raise SkipTest(\"provide tests for Keyring which interfaces keyring module\")\n\n\ndef _cred1_adapter(composite, user=None, password=None):\n    \"\"\"Just a sample adapter from one user/pw type to another\"\"\"\n    return dict(user=user + \"_1\", password=password + \"_2\")\n\n\nclass _CCred1(CompositeCredential):\n    \"\"\"A Simple composite credential which will do some entries transformation\n    \"\"\"\n    _CREDENTIAL_CLASSES = (UserPassword, UserPassword)\n    _CREDENTIAL_ADAPTERS = (_cred1_adapter,)\n\n\n@with_testsui(responses=['user1', 'password1',\n                         'user2', 'password2'])\ndef test_composite_credential1():\n    # basic test of composite credential\n    keyring = MemoryKeyring()\n    cred = _CCred1(\"name\", keyring=keyring)\n    # When queried, does the chain\n    assert_equal(cred(), {'user': 'user1_1', 'password': 'password1_2'})\n    # But the \"Front\" credential is exposed to the user\n    assert_equal(cred.get('user'), 'user1')\n    assert_equal(keyring.get('name', 'user'), 'user1')\n    assert_raises(ValueError, cred.get, 'unknown_field')\n    assert_equal(cred.get('password'), 'password1')\n    assert_equal(keyring.get('name', 'password'), 'password1')\n    # ATM composite credential stores \"derived\" ones unconditionally in the\n    # keyring as well\n    assert_equal(keyring.get('name:1', 'user'), 'user1_1')\n    assert_equal(keyring.get('name:1', 'password'), 'password1_2')\n\n    # and now enter new should remove \"derived\" entries\n    cred.enter_new()\n    assert_equal(keyring.get('name', 'user'), 'user2')\n    assert_equal(keyring.get('name', 'password'), 'password2')\n    # we immediately refresh all credentials in the chain\n    assert_equal(keyring.get('name:1', 'user'), 'user2_1')\n    assert_equal(keyring.get('name:1', 'password'), 'password2_2')\n    assert_equal(cred(), {'user': 'user2_1', 'password': 'password2_2'})\n\n\ndef test_credentials_from_env():\n    keyring = Keyring()\n    cred = AWS_S3(\"test-s3\", keyring=keyring)\n    assert_false(cred.is_known)\n    assert_equal(cred.get('key_id'), None)\n    assert_equal(cred.get('secret_id'), None)\n\n    def _check1():\n        assert_equal(cred.get('key_id'), '1')\n        assert_false(cred.is_known)\n\n    def _check2():\n        assert_equal(cred.get('key_id'), '1')\n        assert_equal(cred.get('secret_id'), '2')\n        assert_true(cred.is_known)\n\n    # this is the old way, should still work\n    with patch.dict('os.environ', {'DATALAD_test_s3_key_id': '1'}):\n        _check1()\n        with patch.dict('os.environ', {'DATALAD_test_s3_secret_id': '2'}):\n            _check2()\n        assert_false(cred.is_known)  # no memory of the past\n\n    # here is the new way\n    import datalad\n    try:\n        with patch.dict('os.environ', {'DATALAD_CREDENTIAL_test__s3_key__id': '1'}):\n            datalad.cfg.reload()\n            _check1()\n            with patch.dict('os.environ', {'DATALAD_CREDENTIAL_test__s3_secret__id': '2'}):\n                datalad.cfg.reload()\n                _check2()\n            datalad.cfg.reload()\n            assert_false(cred.is_known)  # no memory of the past\n    finally:\n        datalad.cfg.reload()\n",
            "new_file_content": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Tests for credentials\"\"\"\n\nfrom unittest.mock import patch\nfrom datalad.tests.utils import (\n    assert_equal,\n    assert_false,\n    assert_in,\n    assert_raises,\n    assert_true,\n    SkipTest,\n    with_testsui,\n)\nfrom datalad.support.keyring_ import (\n    Keyring,\n    MemoryKeyring,\n)\nfrom ..credentials import (\n    AWS_S3,\n    CompositeCredential,\n    UserPassword,\n)\nfrom datalad import cfg as dlcfg\n\n\n@with_testsui(responses=[\n    'user1', 'password1',\n    # when we do provide user to enter_new\n    'newpassword',\n])\ndef test_cred1_enter_new():\n    keyring = MemoryKeyring()\n    cred = UserPassword(\"name\", keyring=keyring)\n    assert_false(cred.is_known)\n    assert_equal(cred.enter_new(), None)\n    assert_true(cred.is_known)\n    assert_equal(keyring.get('name', 'user'), 'user1')\n    assert_equal(keyring.get('name', 'password'), 'password1')\n    keyring.delete('name')\n    assert_raises(KeyError, keyring.delete, 'name', 'user')\n    assert_raises(KeyError, keyring.delete, 'name')\n    assert_equal(keyring.get('name', 'user'), None)\n\n    # Test it blowing up if we provide unknown field\n    with assert_raises(ValueError) as cme:\n        cred.enter_new(username='user')\n    assert_in('field(s): username.  Known but not specified: password, user',\n              str(cme.exception))\n\n    # Test that if user is provided, it is not asked\n    cred.enter_new(user='user2')\n    assert_equal(keyring.get('name', 'user'), 'user2')\n    assert_equal(keyring.get('name', 'password'), 'newpassword')\n\n@with_testsui(responses=['password1', 'newuser', 'newpassword'])\ndef test_cred1_call():\n    keyring = MemoryKeyring()\n    cred = UserPassword(\"name\", keyring=keyring)\n    # we will set the name but not the password, expecting UI\n    # requesting it\n    assert_equal(keyring.set('name', 'user', 'user1'), None)\n    assert_equal(keyring.get('name', 'user'), 'user1')\n    assert_equal(cred(), {'user': 'user1', 'password': 'password1'})\n    assert_equal(keyring.get('name', 'password'), 'password1')\n    # without intervention the same credentials will be reused\n    # in subsequent attempts\n    assert_equal(cred(), {'user': 'user1', 'password': 'password1'})\n    with patch.dict(dlcfg._merged_store, {'datalad.credentials.force-ask': 'yes'}):\n        assert_equal(cred(), {'user': 'newuser', 'password': 'newpassword'})\n    assert_equal(keyring.get('name', 'user'), 'newuser')\n    assert_equal(keyring.get('name', 'password'), 'newpassword')\n\n\ndef test_keyring():\n    # mock out keyring methods and test that we are providing correct values\n    # with 'datalad-' prefix\n    raise SkipTest(\"provide tests for Keyring which interfaces keyring module\")\n\n\ndef _cred1_adapter(composite, user=None, password=None):\n    \"\"\"Just a sample adapter from one user/pw type to another\"\"\"\n    return dict(user=user + \"_1\", password=password + \"_2\")\n\n\nclass _CCred1(CompositeCredential):\n    \"\"\"A Simple composite credential which will do some entries transformation\n    \"\"\"\n    _CREDENTIAL_CLASSES = (UserPassword, UserPassword)\n    _CREDENTIAL_ADAPTERS = (_cred1_adapter,)\n\n\n@with_testsui(responses=['user1', 'password1',\n                         'user2', 'password2'])\ndef test_composite_credential1():\n    # basic test of composite credential\n    keyring = MemoryKeyring()\n    cred = _CCred1(\"name\", keyring=keyring)\n    # When queried, does the chain\n    assert_equal(cred(), {'user': 'user1_1', 'password': 'password1_2'})\n    # But the \"Front\" credential is exposed to the user\n    assert_equal(cred.get('user'), 'user1')\n    assert_equal(keyring.get('name', 'user'), 'user1')\n    assert_raises(ValueError, cred.get, 'unknown_field')\n    assert_equal(cred.get('password'), 'password1')\n    assert_equal(keyring.get('name', 'password'), 'password1')\n    # ATM composite credential stores \"derived\" ones unconditionally in the\n    # keyring as well\n    assert_equal(keyring.get('name:1', 'user'), 'user1_1')\n    assert_equal(keyring.get('name:1', 'password'), 'password1_2')\n\n    # and now enter new should remove \"derived\" entries\n    cred.enter_new()\n    assert_equal(keyring.get('name', 'user'), 'user2')\n    assert_equal(keyring.get('name', 'password'), 'password2')\n    # we immediately refresh all credentials in the chain\n    assert_equal(keyring.get('name:1', 'user'), 'user2_1')\n    assert_equal(keyring.get('name:1', 'password'), 'password2_2')\n    assert_equal(cred(), {'user': 'user2_1', 'password': 'password2_2'})\n\n\ndef test_credentials_from_env():\n    keyring = Keyring()\n    cred = AWS_S3(\"test-s3\", keyring=keyring)\n    assert_false(cred.is_known)\n    assert_equal(cred.get('key_id'), None)\n    assert_equal(cred.get('secret_id'), None)\n\n    def _check1():\n        assert_equal(cred.get('key_id'), '1')\n        assert_false(cred.is_known)\n\n    def _check2():\n        assert_equal(cred.get('key_id'), '1')\n        assert_equal(cred.get('secret_id'), '2')\n        assert_true(cred.is_known)\n\n    # this is the old way, should still work\n    with patch.dict('os.environ', {'DATALAD_test_s3_key_id': '1'}):\n        _check1()\n        with patch.dict('os.environ', {'DATALAD_test_s3_secret_id': '2'}):\n            _check2()\n        assert_false(cred.is_known)  # no memory of the past\n\n    # here is the new way\n    import datalad\n    try:\n        with patch.dict('os.environ', {'DATALAD_CREDENTIAL_test__s3_key__id': '1'}):\n            datalad.cfg.reload()\n            _check1()\n            with patch.dict('os.environ', {'DATALAD_CREDENTIAL_test__s3_secret__id': '2'}):\n                datalad.cfg.reload()\n                _check2()\n            datalad.cfg.reload()\n            assert_false(cred.is_known)  # no memory of the past\n    finally:\n        datalad.cfg.reload()\n",
            "header": {
                "file": {
                    "path": "datalad/downloaders/tests/test_credentials.py"
                },
                "misc_line": null
            },
            "index_line": {
                "old_commit_hash": "b5ccbbe7a",
                "new_commit_hash": "bb6fe5563",
                "mode": "100644"
            },
            "is_binary_file": false,
            "binary_line": null,
            "minus_file": {
                "path": "a/datalad/downloaders/tests/test_credentials.py"
            },
            "plus_file": {
                "path": "b/datalad/downloaders/tests/test_credentials.py"
            },
            "hunks": [
                {
                    "descriptor": {
                        "old_range": {
                            "start": 27,
                            "length": 7
                        },
                        "new_range": {
                            "start": 27,
                            "length": 7
                        },
                        "section": "from ..credentials import ("
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "    CompositeCredential,",
                                "type": "context"
                            },
                            {
                                "content": "    UserPassword,",
                                "type": "context"
                            },
                            {
                                "content": ")",
                                "type": "context"
                            },
                            {
                                "content": "from datalad.config import ConfigManager",
                                "type": "deleted"
                            },
                            {
                                "content": "from datalad import cfg as dlcfg",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "@with_testsui(responses=[",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [],
                    "added_entities": [
                        {
                            "file_name": "datalad/downloaders/tests/test_credentials.py",
                            "type": "statement",
                            "name": "",
                            "content": "from datalad import cfg as dlcfg",
                            "ast_type_str": "ImportFrom",
                            "start_lineno": 30,
                            "end_lineno": 30,
                            "parent": null
                        }
                    ],
                    "deleted_entities": [
                        {
                            "file_name": "datalad/downloaders/tests/test_credentials.py",
                            "type": "statement",
                            "name": "",
                            "content": "from datalad.config import ConfigManager",
                            "ast_type_str": "ImportFrom",
                            "start_lineno": 30,
                            "end_lineno": 30,
                            "parent": null
                        }
                    ]
                },
                {
                    "descriptor": {
                        "old_range": {
                            "start": 59,
                            "length": 7
                        },
                        "new_range": {
                            "start": 59,
                            "length": 7
                        },
                        "section": "def test_cred1_enter_new():"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "    assert_equal(keyring.get('name', 'user'), 'user2')",
                                "type": "context"
                            },
                            {
                                "content": "    assert_equal(keyring.get('name', 'password'), 'newpassword')",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "@with_testsui(responses=['password1'])",
                                "type": "deleted"
                            },
                            {
                                "content": "@with_testsui(responses=['password1', 'newuser', 'newpassword'])",
                                "type": "added"
                            },
                            {
                                "content": "def test_cred1_call():",
                                "type": "context"
                            },
                            {
                                "content": "    keyring = MemoryKeyring()",
                                "type": "context"
                            },
                            {
                                "content": "    cred = UserPassword(\"name\", keyring=keyring)",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [],
                    "added_entities": [],
                    "deleted_entities": []
                },
                {
                    "descriptor": {
                        "old_range": {
                            "start": 69,
                            "length": 6
                        },
                        "new_range": {
                            "start": 69,
                            "length": 13
                        },
                        "section": "def test_cred1_call():"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "    assert_equal(keyring.get('name', 'user'), 'user1')",
                                "type": "context"
                            },
                            {
                                "content": "    assert_equal(cred(), {'user': 'user1', 'password': 'password1'})",
                                "type": "context"
                            },
                            {
                                "content": "    assert_equal(keyring.get('name', 'password'), 'password1')",
                                "type": "context"
                            },
                            {
                                "content": "    # without intervention the same credentials will be reused",
                                "type": "added"
                            },
                            {
                                "content": "    # in subsequent attempts",
                                "type": "added"
                            },
                            {
                                "content": "    assert_equal(cred(), {'user': 'user1', 'password': 'password1'})",
                                "type": "added"
                            },
                            {
                                "content": "    with patch.dict(dlcfg._merged_store, {'datalad.credentials.force-ask': 'yes'}):",
                                "type": "added"
                            },
                            {
                                "content": "        assert_equal(cred(), {'user': 'newuser', 'password': 'newpassword'})",
                                "type": "added"
                            },
                            {
                                "content": "    assert_equal(keyring.get('name', 'user'), 'newuser')",
                                "type": "added"
                            },
                            {
                                "content": "    assert_equal(keyring.get('name', 'password'), 'newpassword')",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "def test_keyring():",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [
                        {
                            "file_name": "datalad/downloaders/tests/test_credentials.py",
                            "type": "function",
                            "name": "test_cred1_call",
                            "content": "def test_cred1_call():\n    keyring = MemoryKeyring()\n    cred = UserPassword(\"name\", keyring=keyring)\n    # we will set the name but not the password, expecting UI\n    # requesting it\n    assert_equal(keyring.set('name', 'user', 'user1'), None)\n    assert_equal(keyring.get('name', 'user'), 'user1')\n    assert_equal(cred(), {'user': 'user1', 'password': 'password1'})\n    assert_equal(keyring.get('name', 'password'), 'password1')\n    # without intervention the same credentials will be reused\n    # in subsequent attempts\n    assert_equal(cred(), {'user': 'user1', 'password': 'password1'})\n    with patch.dict(dlcfg._merged_store, {'datalad.credentials.force-ask': 'yes'}):\n        assert_equal(cred(), {'user': 'newuser', 'password': 'newpassword'})\n    assert_equal(keyring.get('name', 'user'), 'newuser')\n    assert_equal(keyring.get('name', 'password'), 'newpassword')",
                            "ast_type_str": "FunctionDef",
                            "start_lineno": 63,
                            "end_lineno": 78,
                            "parent": null
                        }
                    ],
                    "added_entities": [],
                    "deleted_entities": []
                }
            ]
        },
        {
            "old_file_content": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Common configuration options\n\n\"\"\"\n\n__docformat__ = 'restructuredtext'\n\nfrom appdirs import AppDirs\nfrom os import environ\nfrom os.path import join as opj, expanduser\nfrom datalad.support.constraints import EnsureBool\nfrom datalad.support.constraints import EnsureInt\nfrom datalad.support.constraints import EnsureNone\nfrom datalad.support.constraints import EnsureChoice\nfrom datalad.support.constraints import EnsureListOf\nfrom datalad.support.constraints import EnsureStr\nfrom datalad.utils import on_windows\n\ndirs = AppDirs(\"datalad\", \"datalad.org\")\n\nsubst_rule_docs = \"\"\"\\\nA substitution specification is a string with a match and substitution\nexpression, each following Python's regular expression syntax. Both expressions\nare concatenated to a single string with an arbitrary delimiter character. The\ndelimiter is defined by prefixing the string with the delimiter. Prefix and\ndelimiter are stripped from the expressions (Example:\n\",^http://(.*)$,https://\\\\1\").  This setting can be defined multiple times.\nSubstitutions will be applied incrementally, in order of their definition. The\nfirst substitution in such a series must match, otherwise no further\nsubstitutions in a series will be considered. However, following the first\nmatch all further substitutions in a series are processed, regardless whether\nintermediate expressions match or not.\"\"\"\n\ndefinitions = {\n    'datalad.clone.url-substitute.github': {\n        'ui': ('question', {\n               'title': 'GitHub URL substitution rule',\n               'text': 'Mangling for GitHub-related URL. ' + subst_rule_docs\n        }),\n        'destination': 'global',\n        'default': (\n            # take any github project URL apart into <org>###<identifier>\n            r',https?://github.com/([^/]+)/(.*)$,\\1###\\2',\n            # replace any (back)slashes with a single dash\n            r',[/\\\\]+,-',\n            # replace any whitespace (include urlquoted variant)\n            # with a single underscore\n            r',\\s+|(%2520)+|(%20)+,_',\n            # rebuild functional project URL\n            r',([^#]+)###(.*),https://github.com/\\1/\\2',\n        )\n    },\n    # TODO this one should migrate to the datalad-osf extension. however, right\n    # now extensions cannot provide default configuration\n    # https://github.com/datalad/datalad/issues/5769\n    'datalad.clone.url-substitute.osf': {\n        'ui': ('question', {\n               'title': 'Open Science Framework URL substitution rule',\n               'text': 'Mangling for OSF-related URLs. ' + subst_rule_docs\n        }),\n        'destination': 'global',\n        'default': (\n            # accept browser-provided URL and convert to those accepted by\n            # the datalad-osf extension\n            r',^https://osf.io/([^/]+)[/]*$,osf://\\1',\n        )\n    },\n    # this is actually used in downloaders, but kept cfg name original\n    'datalad.crawl.cache': {\n        'ui': ('yesno', {\n               'title': 'Crawler download caching',\n               'text': 'Should the crawler cache downloaded files?'}),\n        'destination': 'local',\n        'type': bool,\n    },\n    'datalad.externals.nda.dbserver': {\n        'ui': ('question', {\n               'title': 'NDA database server',\n               'text': 'Hostname of the database server'}),\n        'destination': 'global',\n        # Development one is https://development.nimhda.org\n        'default': 'https://nda.nih.gov/DataManager/dataManager',\n    },\n    'datalad.locations.cache': {\n        'ui': ('question', {\n               'title': 'Cache directory',\n               'text': 'Where should datalad cache files?'}),\n        'destination': 'global',\n        'default_fn': lambda: dirs.user_cache_dir,\n    },\n    'datalad.locations.default-dataset': {\n        'ui': ('question', {\n               'title': 'Default dataset path',\n               'text': 'Where should datalad should look for (or install) a '\n                       'default dataset?'}),\n        'destination': 'global',\n        'default_fn': lambda: opj(expanduser('~'), 'datalad'),\n    },\n    'datalad.locations.locks': {\n        'ui': ('question', {\n               'title': 'Lockfile directory',\n               'text': 'Where should datalad store lock files?'}),\n        'destination': 'global',\n        'default_fn': lambda: opj(dirs.user_cache_dir, 'locks')\n    },\n    'datalad.locations.sockets': {\n        'ui': ('question', {\n               'title': 'Socket directory',\n               'text': 'Where should datalad store socket files?'}),\n        'destination': 'global',\n        'default_fn': lambda: opj(dirs.user_cache_dir, 'sockets'),\n    },\n    'datalad.locations.system-procedures': {\n        'ui': ('question', {\n               'title': 'System procedure directory',\n               'text': 'Where should datalad search for system procedures?'}),\n        'destination': 'global',\n        'default_fn': lambda: opj(dirs.site_config_dir, 'procedures'),\n    },\n    'datalad.locations.user-procedures': {\n        'ui': ('question', {\n               'title': 'User procedure directory',\n               'text': 'Where should datalad search for user procedures?'}),\n        'destination': 'global',\n        'default_fn': lambda: opj(dirs.user_config_dir, 'procedures'),\n    },\n    'datalad.locations.extra-procedures': {\n        'ui': ('question', {\n            'title': 'Extra procedure directory',\n            'text': 'Where should datalad search for some additional procedures?'}),\n        'destination': 'global',\n    },\n    'datalad.locations.dataset-procedures': {\n        'ui': ('question', {\n               'title': 'Dataset procedure directory',\n               'text': 'Where should datalad search for dataset procedures (relative to a dataset root)?'}),\n        'destination': 'dataset',\n        'default': opj('.datalad', 'procedures'),\n    },\n    'datalad.exc.str.tblimit': {\n        'ui': ('question', {\n               'title': 'This flag is used by datalad to cap the number of traceback steps included in exception logging and result reporting to DATALAD_EXC_STR_TBLIMIT of pre-processed entries from traceback.'}),\n    },\n    'datalad.fake-dates': {\n        'ui': ('yesno', {\n               'title': 'Fake (anonymize) dates',\n               'text': 'Should the dates in the logs be faked?'}),\n        'destination': 'local',\n        'type': EnsureBool(),\n        'default': False,\n    },\n    'datalad.fake-dates-start': {\n        'ui': ('question', {\n            'title': 'Initial fake date',\n            'text': 'When faking dates and there are no commits in any local branches, generate the date by adding one second to this value (Unix epoch time). The value must be positive.'}),\n        'type': EnsureInt(),\n        'default': 1112911993,\n    },\n    'datalad.github.token-note': {\n        'ui': ('question', {\n            'title': 'Github token note',\n            'text': 'Description for a Personal access token to generate.'}),\n        'default': 'DataLad',\n    },\n    'datalad.tests.nonetwork': {\n        'ui': ('yesno', {\n               'title': 'Skips network tests completely if this flag is set Examples include test for s3, git_repositories, openfmri etc'}),\n        'type': EnsureBool(),\n    },\n    'datalad.tests.nonlo': {\n        'ui': ('question', {\n               'title': 'Specifies network interfaces to bring down/up for testing. Currently used by travis.'}),\n    },\n    'datalad.tests.noteardown': {\n        'ui': ('yesno', {\n               'title': 'Does not execute teardown_package which cleans up temp files and directories created by tests if this flag is set'}),\n        'type': EnsureBool(),\n    },\n    'datalad.tests.dataladremote': {\n        'ui': ('yesno', {\n               'title': 'Binary flag to specify whether each annex repository should get datalad special remote in every test repository'}),\n        'type': EnsureBool(),\n    },\n    'datalad.tests.runcmdline': {\n        'ui': ('yesno', {\n               'title': 'Binary flag to specify if shell testing using shunit2 to be carried out'}),\n        'type': EnsureBool(),\n    },\n    'datalad.tests.ssh': {\n        'ui': ('yesno', {\n               'title': 'Skips SSH tests if this flag is **not** set'}),\n        'type': EnsureBool(),\n    },\n    'datalad.tests.knownfailures.skip': {\n        'ui': ('yesno', {\n               'title': 'Skips tests that are known to currently fail'}),\n        'type': EnsureBool(),\n        'default': True,\n    },\n    'datalad.tests.knownfailures.probe': {\n        'ui': ('yesno', {\n               'title': 'Probes tests that are known to fail on whether or not they are actually still failing'}),\n        'type': EnsureBool(),\n        'default': False,\n    },\n    'datalad.tests.setup.testrepos': {\n        'ui': ('question', {\n            'title': 'Pre-creates repositories for @with_testrepos within setup_package'}),\n        'type': EnsureBool(),\n        'default': False,\n    },\n    'datalad.tests.temp.dir': {\n        'ui': ('question', {\n               'title': 'Create a temporary directory at location specified by this flag. It is used by tests to create a temporary git directory while testing git annex archives etc'}),\n        'type': EnsureStr(),\n        'default_fn': lambda: environ.get('TMPDIR'),\n    },\n    'datalad.tests.temp.keep': {\n        'ui': ('yesno', {\n               'title': 'Function rmtemp will not remove temporary file/directory created for testing if this flag is set'}),\n        'type': EnsureBool(),\n    },\n    'datalad.tests.temp.fs': {\n        'ui': ('question', {\n               'title': 'Specify the temporary file system to use as loop device for testing DATALAD_TESTS_TEMP_DIR creation'}),\n    },\n    'datalad.tests.temp.fssize': {\n        'ui': ('question', {\n               'title': 'Specify the size of temporary file system to use as loop device for testing DATALAD_TESTS_TEMP_DIR creation'}),\n    },\n    'datalad.tests.ui.backend': {\n        'ui': ('question', {\n            'title': 'Tests UI backend',\n            # XXX we could add choices...\n            'text': 'Which UI backend to use'}),\n        'default': 'tests-noninteractive',\n    },\n    'datalad.tests.usecassette': {\n        'ui': ('question', {\n               'title': 'Specifies the location of the file to record network transactions by the VCR module. Currently used by when testing custom special remotes'}),\n    },\n    'datalad.tests.cache': {\n        'ui': ('question', {\n            'title': 'Cache directory for tests',\n            'text': 'Where should datalad cache test files?'}),\n        'destination': 'global',\n        'default_fn': lambda: opj(dirs.user_cache_dir, 'tests')\n    },\n    'datalad.log.level': {\n        'ui': ('question', {\n            'title': 'Used for control the verbosity of logs printed to '\n                     'stdout while running datalad commands/debugging'}),\n    },\n    'datalad.log.result-level': {\n        'ui': ('question', {\n               'title': 'Log level for command result messages',\n               'text': \"Overrides the default behavior of logging 'impossible' \"\n                       \"results as a warning, 'error' results as errors, and \"\n                       \"everything else as 'debug' with a single alternative \"\n                       \"log level\"}),\n        'type': EnsureChoice('debug', 'info', 'warning', 'error'),\n        # None keeps the default behavior\n        'default': 'None',\n    },\n    'datalad.log.name': {\n        'ui': ('question', {\n            'title': 'Include name of the log target in the log line'}),\n    },\n    'datalad.log.names': {\n        'ui': ('question', {\n            'title': 'Which names (,-separated) to print log lines for'}),\n    },\n    'datalad.log.namesre': {\n        'ui': ('question', {\n            'title': 'Regular expression for which names to print log lines for'}),\n    },\n    'datalad.log.outputs': {\n        'ui': ('question', {\n               'title': 'Whether to log stdout and stderr for executed commands',\n               'text': 'When enabled, setting the log level to 5 '\n                       'should catch all execution output, '\n                       'though some output may be logged at higher levels'}),\n        'default': False,\n        'type': EnsureBool(),\n    },\n    'datalad.log.timestamp': {\n        'ui': ('yesno', {\n               'title': 'Used to add timestamp to datalad logs'}),\n        'default': False,\n        'type': EnsureBool(),\n    },\n    'datalad.log.traceback': {\n        'ui': ('question', {\n               'title': 'Runs TraceBack function with collide set to True, if this flag is set to \"collide\". This replaces any common prefix between current traceback log and previous invocation with \"...\"'}),\n    },\n    'datalad.log.exc': {\n        'ui': ('yesno', {\n               'title': 'Include exceptions and their traceback in log messages. If set, \\'datalad.exc.str.tblimit\\' applies.'}),\n        'default': False,\n        'type': EnsureBool(),\n    },\n    'datalad.ssh.identityfile': {\n        'ui': ('question', {\n               'title': \"If set, pass this file as ssh's -i option.\"}),\n        'destination': 'global',\n        'default': None,\n    },\n    'datalad.ssh.multiplex-connections': {\n        'ui': ('question', {\n               'title': \"Whether to use a single shared connection for multiple SSH processes aiming at the same target.\"}),\n        'destination': 'global',\n        'default': not on_windows,\n        'type': EnsureBool(),\n    },\n    'datalad.annex.retry': {\n        'ui': ('question',\n               {'title': 'Value for annex.retry to use for git-annex calls',\n                'text': 'On transfer failure, annex.retry (sans \"datalad.\") '\n                        'controls the number of times that git-annex retries. '\n                        'DataLad will call git-annex with annex.retry set '\n                        'to the value here unless the annex.retry '\n                        'is explicitly configured'}),\n        'type': EnsureInt(),\n        'default': 3,\n    },\n    'datalad.repo.backend': {\n        'ui': ('question', {\n               'title': 'git-annex backend',\n               'text': 'Backend to use when creating git-annex repositories'}),\n        'default': 'MD5E',\n    },\n    'datalad.repo.direct': {\n        'ui': ('yesno', {\n               'title': 'Direct Mode for git-annex repositories',\n               'text': 'Set this flag to create annex repositories in direct mode by default'}),\n        'type': EnsureBool(),\n        'default': False,\n    },\n    'datalad.repo.version': {\n        'ui': ('question', {\n               'title': 'git-annex repository version',\n               'text': 'Specifies the repository version for git-annex to be used by default'}),\n        'type': EnsureInt(),\n        'default': 8,\n    },\n    'datalad.metadata.maxfieldsize': {\n        'ui': ('question', {\n               'title': 'Maximum metadata field size',\n               'text': 'Metadata fields exceeding this size (in bytes/chars) are excluded from metadata extractio'}),\n        'default': 100000,\n        'type': EnsureInt(),\n    },\n    'datalad.metadata.nativetype': {\n        'ui': ('question', {\n               'title': 'Native dataset metadata scheme',\n               'text': 'Set this label to engage a particular metadata extraction parser'}),\n    },\n    'datalad.metadata.store-aggregate-content': {\n        'ui': ('question', {\n               'title': 'Aggregated content metadata storage',\n               'text': 'If this flag is enabled, content metadata is aggregated into superdataset to allow for discovery of individual files. If disable unique content metadata values are still aggregated to enable dataset discovery'}),\n        'type': EnsureBool(),\n        'default': True,\n    },\n    'datalad.search.default-mode': {\n        'ui': ('question', {\n               'title': 'Default search mode',\n               'text': 'Label of the mode to be used by default'}),\n        'type': EnsureChoice('egrep', 'textblob', 'autofield'),  # graph,...\n        'default': 'egrep',\n    },\n    'datalad.search.index-default-documenttype': {\n        'ui': ('question', {\n               'title': 'Type of search index documents',\n               'text': 'Labels of document types to include in a default search index'}),\n        'type': EnsureChoice('all', 'datasets', 'files'),\n        'default': 'datasets',\n    },\n    'datalad.metadata.create-aggregate-annex-limit': {\n        'ui': ('question', {\n               'title': 'Limit configuration annexing aggregated metadata in new dataset',\n               'text': 'Git-annex large files expression (see https://git-annex.branchable.com/tips/largefiles; given expression will be wrapped in parentheses)'}),\n        'default': 'anything',\n    },\n    'datalad.runtime.max-annex-jobs': {\n        'ui': ('question', {\n               'title': 'Maximum number of git-annex jobs to request when \"jobs\" option set to \"auto\" (default)',\n               'text': 'Set this value to enable parallel annex jobs that may speed up certain operations (e.g. get file content). The effective number of jobs will not exceed the number of available CPU cores (or 3 if there is less than 3 cores).'}),\n        'type': EnsureInt(),\n        'default': 1,\n    },\n    'datalad.runtime.max-jobs': {\n        'ui': ('question', {\n            'title': 'Maximum number of jobs DataLad can run in \"parallel\"',\n            'text': 'Set this value to enable parallel multi-threaded DataLad jobs that may speed up certain '\n                    'operations, in particular operation across multiple datasets (e.g., install multiple '\n                    'subdatasets, etc).'}),\n        'type': EnsureInt(),\n        'default': 1,\n    },\n    'datalad.runtime.raiseonerror': {\n        'ui': ('question', {\n               'title': 'Error behavior',\n               'text': 'Set this flag to cause DataLad to raise an exception on errors that would have otherwise just get logged'}),\n        'type': EnsureBool(),\n        'default': False,\n    },\n    'datalad.runtime.report-status': {\n        'ui': ('question', {\n               'title': 'Command line result reporting behavior',\n               'text': \"If set (to other than 'all'), constrains command result report to records matching the given status. 'success' is a synonym for 'ok' OR 'notneeded', 'failure' stands for 'impossible' OR 'error'\"}),\n        'type': EnsureChoice('all', 'success', 'failure', 'ok', 'notneeded', 'impossible', 'error'),\n        'default': None,\n    },\n    'datalad.runtime.stalled-external': {\n        'ui': ('question', {\n            'title': 'Behavior for handing external processes',\n            'text': 'What to do with external processes if they do not finish in some minimal reasonable time. '\n                    'If \"abandon\", datalad would proceed without waiting for external process to exit. '\n                    'ATM applies only to batched git-annex processes. Should be changed with caution.'}),\n        'type': EnsureChoice('wait', 'abandon'),\n        'default': 'wait',\n    },\n    'datalad.search.indexercachesize': {\n        'ui': ('question', {\n               'title': 'Maximum cache size for search index (per process)',\n               'text': 'Actual memory consumption can be twice as high as this value in MB (one process per CPU is used)'}),\n        'default': 256,\n        'type': EnsureInt(),\n    },\n    'datalad.ui.progressbar': {\n        'ui': ('question', {\n            'title': 'UI progress bars',\n            'text': 'Default backend for progress reporting'}),\n        'default': None,\n        'type': EnsureChoice('tqdm', 'tqdm-ipython', 'log', 'none'),\n    },\n    'datalad.ui.color': {\n        'ui': ('question', {\n            'title': 'Colored terminal output',\n            'text': 'Enable or disable ANSI color codes in outputs; \"on\" overrides NO_COLOR environment variable'}),\n        'default': 'auto',\n        'type': EnsureChoice('on', 'off', 'auto'),\n    },\n    'datalad.ui.suppress-similar-results': {\n        'ui': ('question', {\n            'title': 'Suppress rendering of similar repetitive results',\n            'text': \"If enabled, after a certain number of subsequent \"\n                    \"results that are identical regarding key properties, \"\n                    \"such as 'status', 'action', and 'type', additional \"\n                    \"similar results are not rendered by the common result \"\n                    \"renderer anymore. Instead, a count \"\n                    \"of suppressed results is displayed. If disabled, or \"\n                    \"when not running in an interactive terminal, all results \"\n                    \"are rendered.\"}),\n        'default': True,\n        'type': EnsureBool(),\n    },\n    'datalad.ui.suppress-similar-results-threshold': {\n        'ui': ('question', {\n            'title': 'Threshold for suppressing similar repetitive results',\n            'text': \"Minimum number of similar results to occur before \"\n                    \"suppression is considered. \"\n                    \"See 'datalad.ui.suppress-similar-results' for more \"\n                    \"information.\"}),\n        'default': 10,\n        'type': EnsureInt(),\n    },\n    'datalad.save.no-message': {\n        'ui': ('question', {\n            'title': 'Commit message handling',\n            'text': 'When no commit message was provided: '\n                    'attempt to obtain one interactively (interactive); '\n                    'or use a generic commit message (generic). '\n                    'NOTE: The interactive option is experimental. The '\n                    'behavior may change in backwards-incompatible ways.'}),\n        'default': 'generic',\n        'type': EnsureChoice('interactive', 'generic'),\n    },\n    'datalad.install.inherit-local-origin': {\n        'ui': ('question', {\n            'title': 'Inherit local origin of dataset source',\n            'text': \"If enabled, a local 'origin' remote of a local dataset \"\n                    \"clone source is configured as an 'origin-2' remote \"\n                    \"to make its annex automatically available. The process \"\n                    \"is repeated recursively for any further qualifying \"\n                    \"'origin' dataset thereof.\"\n                    \"Note that if clone.defaultRemoteName is configured \"\n                    \"to use a name other than 'origin', that name will be \"\n                    \"used instead.\"}),\n        'default': True,\n        'type': EnsureBool(),\n    },\n}\n\n\ndef compute_cfg_defaults():\n    \"\"\"Compute dynamic defaults for configuration options.\n\n    These are options that depend on things like $HOME that change under our\n    testing setup.\n    \"\"\"\n    for key, value in definitions.items():\n        def_fn = value.get(\"default_fn\")\n        if def_fn:\n            value['default'] = def_fn()\n\n\ncompute_cfg_defaults()\n",
            "new_file_content": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Common configuration options\n\n\"\"\"\n\n__docformat__ = 'restructuredtext'\n\nfrom appdirs import AppDirs\nfrom os import environ\nfrom os.path import join as opj, expanduser\nfrom datalad.support.constraints import EnsureBool\nfrom datalad.support.constraints import EnsureInt\nfrom datalad.support.constraints import EnsureNone\nfrom datalad.support.constraints import EnsureChoice\nfrom datalad.support.constraints import EnsureListOf\nfrom datalad.support.constraints import EnsureStr\nfrom datalad.utils import on_windows\n\ndirs = AppDirs(\"datalad\", \"datalad.org\")\n\nsubst_rule_docs = \"\"\"\\\nA substitution specification is a string with a match and substitution\nexpression, each following Python's regular expression syntax. Both expressions\nare concatenated to a single string with an arbitrary delimiter character. The\ndelimiter is defined by prefixing the string with the delimiter. Prefix and\ndelimiter are stripped from the expressions (Example:\n\",^http://(.*)$,https://\\\\1\").  This setting can be defined multiple times.\nSubstitutions will be applied incrementally, in order of their definition. The\nfirst substitution in such a series must match, otherwise no further\nsubstitutions in a series will be considered. However, following the first\nmatch all further substitutions in a series are processed, regardless whether\nintermediate expressions match or not.\"\"\"\n\ndefinitions = {\n    'datalad.clone.url-substitute.github': {\n        'ui': ('question', {\n               'title': 'GitHub URL substitution rule',\n               'text': 'Mangling for GitHub-related URL. ' + subst_rule_docs\n        }),\n        'destination': 'global',\n        'default': (\n            # take any github project URL apart into <org>###<identifier>\n            r',https?://github.com/([^/]+)/(.*)$,\\1###\\2',\n            # replace any (back)slashes with a single dash\n            r',[/\\\\]+,-',\n            # replace any whitespace (include urlquoted variant)\n            # with a single underscore\n            r',\\s+|(%2520)+|(%20)+,_',\n            # rebuild functional project URL\n            r',([^#]+)###(.*),https://github.com/\\1/\\2',\n        )\n    },\n    # TODO this one should migrate to the datalad-osf extension. however, right\n    # now extensions cannot provide default configuration\n    # https://github.com/datalad/datalad/issues/5769\n    'datalad.clone.url-substitute.osf': {\n        'ui': ('question', {\n               'title': 'Open Science Framework URL substitution rule',\n               'text': 'Mangling for OSF-related URLs. ' + subst_rule_docs\n        }),\n        'destination': 'global',\n        'default': (\n            # accept browser-provided URL and convert to those accepted by\n            # the datalad-osf extension\n            r',^https://osf.io/([^/]+)[/]*$,osf://\\1',\n        )\n    },\n    # this is actually used in downloaders, but kept cfg name original\n    'datalad.crawl.cache': {\n        'ui': ('yesno', {\n               'title': 'Crawler download caching',\n               'text': 'Should the crawler cache downloaded files?'}),\n        'destination': 'local',\n        'type': bool,\n    },\n    # this is actually used in downloaders, but kept cfg name original\n    'datalad.credentials.force-ask': {\n        'ui': ('yesno', {\n               'title': 'Force (re-)entry of credentials',\n               'text': 'Should DataLad prompt for credential (re-)entry? This '\n                       'can be used to update previously stored credentials.'}),\n        'type': bool,\n        'default': False,\n    },\n    'datalad.externals.nda.dbserver': {\n        'ui': ('question', {\n               'title': 'NDA database server',\n               'text': 'Hostname of the database server'}),\n        'destination': 'global',\n        # Development one is https://development.nimhda.org\n        'default': 'https://nda.nih.gov/DataManager/dataManager',\n    },\n    'datalad.locations.cache': {\n        'ui': ('question', {\n               'title': 'Cache directory',\n               'text': 'Where should datalad cache files?'}),\n        'destination': 'global',\n        'default_fn': lambda: dirs.user_cache_dir,\n    },\n    'datalad.locations.default-dataset': {\n        'ui': ('question', {\n               'title': 'Default dataset path',\n               'text': 'Where should datalad should look for (or install) a '\n                       'default dataset?'}),\n        'destination': 'global',\n        'default_fn': lambda: opj(expanduser('~'), 'datalad'),\n    },\n    'datalad.locations.locks': {\n        'ui': ('question', {\n               'title': 'Lockfile directory',\n               'text': 'Where should datalad store lock files?'}),\n        'destination': 'global',\n        'default_fn': lambda: opj(dirs.user_cache_dir, 'locks')\n    },\n    'datalad.locations.sockets': {\n        'ui': ('question', {\n               'title': 'Socket directory',\n               'text': 'Where should datalad store socket files?'}),\n        'destination': 'global',\n        'default_fn': lambda: opj(dirs.user_cache_dir, 'sockets'),\n    },\n    'datalad.locations.system-procedures': {\n        'ui': ('question', {\n               'title': 'System procedure directory',\n               'text': 'Where should datalad search for system procedures?'}),\n        'destination': 'global',\n        'default_fn': lambda: opj(dirs.site_config_dir, 'procedures'),\n    },\n    'datalad.locations.user-procedures': {\n        'ui': ('question', {\n               'title': 'User procedure directory',\n               'text': 'Where should datalad search for user procedures?'}),\n        'destination': 'global',\n        'default_fn': lambda: opj(dirs.user_config_dir, 'procedures'),\n    },\n    'datalad.locations.extra-procedures': {\n        'ui': ('question', {\n            'title': 'Extra procedure directory',\n            'text': 'Where should datalad search for some additional procedures?'}),\n        'destination': 'global',\n    },\n    'datalad.locations.dataset-procedures': {\n        'ui': ('question', {\n               'title': 'Dataset procedure directory',\n               'text': 'Where should datalad search for dataset procedures (relative to a dataset root)?'}),\n        'destination': 'dataset',\n        'default': opj('.datalad', 'procedures'),\n    },\n    'datalad.exc.str.tblimit': {\n        'ui': ('question', {\n               'title': 'This flag is used by datalad to cap the number of traceback steps included in exception logging and result reporting to DATALAD_EXC_STR_TBLIMIT of pre-processed entries from traceback.'}),\n    },\n    'datalad.fake-dates': {\n        'ui': ('yesno', {\n               'title': 'Fake (anonymize) dates',\n               'text': 'Should the dates in the logs be faked?'}),\n        'destination': 'local',\n        'type': EnsureBool(),\n        'default': False,\n    },\n    'datalad.fake-dates-start': {\n        'ui': ('question', {\n            'title': 'Initial fake date',\n            'text': 'When faking dates and there are no commits in any local branches, generate the date by adding one second to this value (Unix epoch time). The value must be positive.'}),\n        'type': EnsureInt(),\n        'default': 1112911993,\n    },\n    'datalad.github.token-note': {\n        'ui': ('question', {\n            'title': 'Github token note',\n            'text': 'Description for a Personal access token to generate.'}),\n        'default': 'DataLad',\n    },\n    'datalad.tests.nonetwork': {\n        'ui': ('yesno', {\n               'title': 'Skips network tests completely if this flag is set Examples include test for s3, git_repositories, openfmri etc'}),\n        'type': EnsureBool(),\n    },\n    'datalad.tests.nonlo': {\n        'ui': ('question', {\n               'title': 'Specifies network interfaces to bring down/up for testing. Currently used by travis.'}),\n    },\n    'datalad.tests.noteardown': {\n        'ui': ('yesno', {\n               'title': 'Does not execute teardown_package which cleans up temp files and directories created by tests if this flag is set'}),\n        'type': EnsureBool(),\n    },\n    'datalad.tests.dataladremote': {\n        'ui': ('yesno', {\n               'title': 'Binary flag to specify whether each annex repository should get datalad special remote in every test repository'}),\n        'type': EnsureBool(),\n    },\n    'datalad.tests.runcmdline': {\n        'ui': ('yesno', {\n               'title': 'Binary flag to specify if shell testing using shunit2 to be carried out'}),\n        'type': EnsureBool(),\n    },\n    'datalad.tests.ssh': {\n        'ui': ('yesno', {\n               'title': 'Skips SSH tests if this flag is **not** set'}),\n        'type': EnsureBool(),\n    },\n    'datalad.tests.knownfailures.skip': {\n        'ui': ('yesno', {\n               'title': 'Skips tests that are known to currently fail'}),\n        'type': EnsureBool(),\n        'default': True,\n    },\n    'datalad.tests.knownfailures.probe': {\n        'ui': ('yesno', {\n               'title': 'Probes tests that are known to fail on whether or not they are actually still failing'}),\n        'type': EnsureBool(),\n        'default': False,\n    },\n    'datalad.tests.setup.testrepos': {\n        'ui': ('question', {\n            'title': 'Pre-creates repositories for @with_testrepos within setup_package'}),\n        'type': EnsureBool(),\n        'default': False,\n    },\n    'datalad.tests.temp.dir': {\n        'ui': ('question', {\n               'title': 'Create a temporary directory at location specified by this flag. It is used by tests to create a temporary git directory while testing git annex archives etc'}),\n        'type': EnsureStr(),\n        'default_fn': lambda: environ.get('TMPDIR'),\n    },\n    'datalad.tests.temp.keep': {\n        'ui': ('yesno', {\n               'title': 'Function rmtemp will not remove temporary file/directory created for testing if this flag is set'}),\n        'type': EnsureBool(),\n    },\n    'datalad.tests.temp.fs': {\n        'ui': ('question', {\n               'title': 'Specify the temporary file system to use as loop device for testing DATALAD_TESTS_TEMP_DIR creation'}),\n    },\n    'datalad.tests.temp.fssize': {\n        'ui': ('question', {\n               'title': 'Specify the size of temporary file system to use as loop device for testing DATALAD_TESTS_TEMP_DIR creation'}),\n    },\n    'datalad.tests.ui.backend': {\n        'ui': ('question', {\n            'title': 'Tests UI backend',\n            # XXX we could add choices...\n            'text': 'Which UI backend to use'}),\n        'default': 'tests-noninteractive',\n    },\n    'datalad.tests.usecassette': {\n        'ui': ('question', {\n               'title': 'Specifies the location of the file to record network transactions by the VCR module. Currently used by when testing custom special remotes'}),\n    },\n    'datalad.tests.cache': {\n        'ui': ('question', {\n            'title': 'Cache directory for tests',\n            'text': 'Where should datalad cache test files?'}),\n        'destination': 'global',\n        'default_fn': lambda: opj(dirs.user_cache_dir, 'tests')\n    },\n    'datalad.log.level': {\n        'ui': ('question', {\n            'title': 'Used for control the verbosity of logs printed to '\n                     'stdout while running datalad commands/debugging'}),\n    },\n    'datalad.log.result-level': {\n        'ui': ('question', {\n               'title': 'Log level for command result messages',\n               'text': \"Overrides the default behavior of logging 'impossible' \"\n                       \"results as a warning, 'error' results as errors, and \"\n                       \"everything else as 'debug' with a single alternative \"\n                       \"log level\"}),\n        'type': EnsureChoice('debug', 'info', 'warning', 'error'),\n        # None keeps the default behavior\n        'default': 'None',\n    },\n    'datalad.log.name': {\n        'ui': ('question', {\n            'title': 'Include name of the log target in the log line'}),\n    },\n    'datalad.log.names': {\n        'ui': ('question', {\n            'title': 'Which names (,-separated) to print log lines for'}),\n    },\n    'datalad.log.namesre': {\n        'ui': ('question', {\n            'title': 'Regular expression for which names to print log lines for'}),\n    },\n    'datalad.log.outputs': {\n        'ui': ('question', {\n               'title': 'Whether to log stdout and stderr for executed commands',\n               'text': 'When enabled, setting the log level to 5 '\n                       'should catch all execution output, '\n                       'though some output may be logged at higher levels'}),\n        'default': False,\n        'type': EnsureBool(),\n    },\n    'datalad.log.timestamp': {\n        'ui': ('yesno', {\n               'title': 'Used to add timestamp to datalad logs'}),\n        'default': False,\n        'type': EnsureBool(),\n    },\n    'datalad.log.traceback': {\n        'ui': ('question', {\n               'title': 'Runs TraceBack function with collide set to True, if this flag is set to \"collide\". This replaces any common prefix between current traceback log and previous invocation with \"...\"'}),\n    },\n    'datalad.log.exc': {\n        'ui': ('yesno', {\n               'title': 'Include exceptions and their traceback in log messages. If set, \\'datalad.exc.str.tblimit\\' applies.'}),\n        'default': False,\n        'type': EnsureBool(),\n    },\n    'datalad.ssh.identityfile': {\n        'ui': ('question', {\n               'title': \"If set, pass this file as ssh's -i option.\"}),\n        'destination': 'global',\n        'default': None,\n    },\n    'datalad.ssh.multiplex-connections': {\n        'ui': ('question', {\n               'title': \"Whether to use a single shared connection for multiple SSH processes aiming at the same target.\"}),\n        'destination': 'global',\n        'default': not on_windows,\n        'type': EnsureBool(),\n    },\n    'datalad.annex.retry': {\n        'ui': ('question',\n               {'title': 'Value for annex.retry to use for git-annex calls',\n                'text': 'On transfer failure, annex.retry (sans \"datalad.\") '\n                        'controls the number of times that git-annex retries. '\n                        'DataLad will call git-annex with annex.retry set '\n                        'to the value here unless the annex.retry '\n                        'is explicitly configured'}),\n        'type': EnsureInt(),\n        'default': 3,\n    },\n    'datalad.repo.backend': {\n        'ui': ('question', {\n               'title': 'git-annex backend',\n               'text': 'Backend to use when creating git-annex repositories'}),\n        'default': 'MD5E',\n    },\n    'datalad.repo.direct': {\n        'ui': ('yesno', {\n               'title': 'Direct Mode for git-annex repositories',\n               'text': 'Set this flag to create annex repositories in direct mode by default'}),\n        'type': EnsureBool(),\n        'default': False,\n    },\n    'datalad.repo.version': {\n        'ui': ('question', {\n               'title': 'git-annex repository version',\n               'text': 'Specifies the repository version for git-annex to be used by default'}),\n        'type': EnsureInt(),\n        'default': 8,\n    },\n    'datalad.metadata.maxfieldsize': {\n        'ui': ('question', {\n               'title': 'Maximum metadata field size',\n               'text': 'Metadata fields exceeding this size (in bytes/chars) are excluded from metadata extractio'}),\n        'default': 100000,\n        'type': EnsureInt(),\n    },\n    'datalad.metadata.nativetype': {\n        'ui': ('question', {\n               'title': 'Native dataset metadata scheme',\n               'text': 'Set this label to engage a particular metadata extraction parser'}),\n    },\n    'datalad.metadata.store-aggregate-content': {\n        'ui': ('question', {\n               'title': 'Aggregated content metadata storage',\n               'text': 'If this flag is enabled, content metadata is aggregated into superdataset to allow for discovery of individual files. If disable unique content metadata values are still aggregated to enable dataset discovery'}),\n        'type': EnsureBool(),\n        'default': True,\n    },\n    'datalad.search.default-mode': {\n        'ui': ('question', {\n               'title': 'Default search mode',\n               'text': 'Label of the mode to be used by default'}),\n        'type': EnsureChoice('egrep', 'textblob', 'autofield'),  # graph,...\n        'default': 'egrep',\n    },\n    'datalad.search.index-default-documenttype': {\n        'ui': ('question', {\n               'title': 'Type of search index documents',\n               'text': 'Labels of document types to include in a default search index'}),\n        'type': EnsureChoice('all', 'datasets', 'files'),\n        'default': 'datasets',\n    },\n    'datalad.metadata.create-aggregate-annex-limit': {\n        'ui': ('question', {\n               'title': 'Limit configuration annexing aggregated metadata in new dataset',\n               'text': 'Git-annex large files expression (see https://git-annex.branchable.com/tips/largefiles; given expression will be wrapped in parentheses)'}),\n        'default': 'anything',\n    },\n    'datalad.runtime.max-annex-jobs': {\n        'ui': ('question', {\n               'title': 'Maximum number of git-annex jobs to request when \"jobs\" option set to \"auto\" (default)',\n               'text': 'Set this value to enable parallel annex jobs that may speed up certain operations (e.g. get file content). The effective number of jobs will not exceed the number of available CPU cores (or 3 if there is less than 3 cores).'}),\n        'type': EnsureInt(),\n        'default': 1,\n    },\n    'datalad.runtime.max-jobs': {\n        'ui': ('question', {\n            'title': 'Maximum number of jobs DataLad can run in \"parallel\"',\n            'text': 'Set this value to enable parallel multi-threaded DataLad jobs that may speed up certain '\n                    'operations, in particular operation across multiple datasets (e.g., install multiple '\n                    'subdatasets, etc).'}),\n        'type': EnsureInt(),\n        'default': 1,\n    },\n    'datalad.runtime.raiseonerror': {\n        'ui': ('question', {\n               'title': 'Error behavior',\n               'text': 'Set this flag to cause DataLad to raise an exception on errors that would have otherwise just get logged'}),\n        'type': EnsureBool(),\n        'default': False,\n    },\n    'datalad.runtime.report-status': {\n        'ui': ('question', {\n               'title': 'Command line result reporting behavior',\n               'text': \"If set (to other than 'all'), constrains command result report to records matching the given status. 'success' is a synonym for 'ok' OR 'notneeded', 'failure' stands for 'impossible' OR 'error'\"}),\n        'type': EnsureChoice('all', 'success', 'failure', 'ok', 'notneeded', 'impossible', 'error'),\n        'default': None,\n    },\n    'datalad.runtime.stalled-external': {\n        'ui': ('question', {\n            'title': 'Behavior for handing external processes',\n            'text': 'What to do with external processes if they do not finish in some minimal reasonable time. '\n                    'If \"abandon\", datalad would proceed without waiting for external process to exit. '\n                    'ATM applies only to batched git-annex processes. Should be changed with caution.'}),\n        'type': EnsureChoice('wait', 'abandon'),\n        'default': 'wait',\n    },\n    'datalad.search.indexercachesize': {\n        'ui': ('question', {\n               'title': 'Maximum cache size for search index (per process)',\n               'text': 'Actual memory consumption can be twice as high as this value in MB (one process per CPU is used)'}),\n        'default': 256,\n        'type': EnsureInt(),\n    },\n    'datalad.ui.progressbar': {\n        'ui': ('question', {\n            'title': 'UI progress bars',\n            'text': 'Default backend for progress reporting'}),\n        'default': None,\n        'type': EnsureChoice('tqdm', 'tqdm-ipython', 'log', 'none'),\n    },\n    'datalad.ui.color': {\n        'ui': ('question', {\n            'title': 'Colored terminal output',\n            'text': 'Enable or disable ANSI color codes in outputs; \"on\" overrides NO_COLOR environment variable'}),\n        'default': 'auto',\n        'type': EnsureChoice('on', 'off', 'auto'),\n    },\n    'datalad.ui.suppress-similar-results': {\n        'ui': ('question', {\n            'title': 'Suppress rendering of similar repetitive results',\n            'text': \"If enabled, after a certain number of subsequent \"\n                    \"results that are identical regarding key properties, \"\n                    \"such as 'status', 'action', and 'type', additional \"\n                    \"similar results are not rendered by the common result \"\n                    \"renderer anymore. Instead, a count \"\n                    \"of suppressed results is displayed. If disabled, or \"\n                    \"when not running in an interactive terminal, all results \"\n                    \"are rendered.\"}),\n        'default': True,\n        'type': EnsureBool(),\n    },\n    'datalad.ui.suppress-similar-results-threshold': {\n        'ui': ('question', {\n            'title': 'Threshold for suppressing similar repetitive results',\n            'text': \"Minimum number of similar results to occur before \"\n                    \"suppression is considered. \"\n                    \"See 'datalad.ui.suppress-similar-results' for more \"\n                    \"information.\"}),\n        'default': 10,\n        'type': EnsureInt(),\n    },\n    'datalad.save.no-message': {\n        'ui': ('question', {\n            'title': 'Commit message handling',\n            'text': 'When no commit message was provided: '\n                    'attempt to obtain one interactively (interactive); '\n                    'or use a generic commit message (generic). '\n                    'NOTE: The interactive option is experimental. The '\n                    'behavior may change in backwards-incompatible ways.'}),\n        'default': 'generic',\n        'type': EnsureChoice('interactive', 'generic'),\n    },\n    'datalad.install.inherit-local-origin': {\n        'ui': ('question', {\n            'title': 'Inherit local origin of dataset source',\n            'text': \"If enabled, a local 'origin' remote of a local dataset \"\n                    \"clone source is configured as an 'origin-2' remote \"\n                    \"to make its annex automatically available. The process \"\n                    \"is repeated recursively for any further qualifying \"\n                    \"'origin' dataset thereof.\"\n                    \"Note that if clone.defaultRemoteName is configured \"\n                    \"to use a name other than 'origin', that name will be \"\n                    \"used instead.\"}),\n        'default': True,\n        'type': EnsureBool(),\n    },\n}\n\n\ndef compute_cfg_defaults():\n    \"\"\"Compute dynamic defaults for configuration options.\n\n    These are options that depend on things like $HOME that change under our\n    testing setup.\n    \"\"\"\n    for key, value in definitions.items():\n        def_fn = value.get(\"default_fn\")\n        if def_fn:\n            value['default'] = def_fn()\n\n\ncompute_cfg_defaults()\n",
            "header": {
                "file": {
                    "path": "datalad/interface/common_cfg.py"
                },
                "misc_line": null
            },
            "index_line": {
                "old_commit_hash": "7485cf577",
                "new_commit_hash": "58d9d49a5",
                "mode": "100644"
            },
            "is_binary_file": false,
            "binary_line": null,
            "minus_file": {
                "path": "a/datalad/interface/common_cfg.py"
            },
            "plus_file": {
                "path": "b/datalad/interface/common_cfg.py"
            },
            "hunks": [
                {
                    "descriptor": {
                        "old_range": {
                            "start": 80,
                            "length": 6
                        },
                        "new_range": {
                            "start": 80,
                            "length": 15
                        },
                        "section": "definitions = {"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "        'destination': 'local',",
                                "type": "context"
                            },
                            {
                                "content": "        'type': bool,",
                                "type": "context"
                            },
                            {
                                "content": "    },",
                                "type": "context"
                            },
                            {
                                "content": "    # this is actually used in downloaders, but kept cfg name original",
                                "type": "added"
                            },
                            {
                                "content": "    'datalad.credentials.force-ask': {",
                                "type": "added"
                            },
                            {
                                "content": "        'ui': ('yesno', {",
                                "type": "added"
                            },
                            {
                                "content": "               'title': 'Force (re-)entry of credentials',",
                                "type": "added"
                            },
                            {
                                "content": "               'text': 'Should DataLad prompt for credential (re-)entry? This '",
                                "type": "added"
                            },
                            {
                                "content": "                       'can be used to update previously stored credentials.'}),",
                                "type": "added"
                            },
                            {
                                "content": "        'type': bool,",
                                "type": "added"
                            },
                            {
                                "content": "        'default': False,",
                                "type": "added"
                            },
                            {
                                "content": "    },",
                                "type": "added"
                            },
                            {
                                "content": "    'datalad.externals.nda.dbserver': {",
                                "type": "context"
                            },
                            {
                                "content": "        'ui': ('question', {",
                                "type": "context"
                            },
                            {
                                "content": "               'title': 'NDA database server',",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [],
                    "added_entities": [
                        {
                            "file_name": "datalad/interface/common_cfg.py",
                            "type": "statement",
                            "name": "",
                            "content": "definitions = {\n    'datalad.clone.url-substitute.github': {\n        'ui': ('question', {\n               'title': 'GitHub URL substitution rule',\n               'text': 'Mangling for GitHub-related URL. ' + subst_rule_docs\n        }),\n        'destination': 'global',\n        'default': (\n            # take any github project URL apart into <org>###<identifier>\n            r',https?://github.com/([^/]+)/(.*)$,\\1###\\2',\n            # replace any (back)slashes with a single dash\n            r',[/\\\\]+,-',\n            # replace any whitespace (include urlquoted variant)\n            # with a single underscore\n            r',\\s+|(%2520)+|(%20)+,_',\n            # rebuild functional project URL\n            r',([^#]+)###(.*),https://github.com/\\1/\\2',\n        )\n    },\n    # TODO this one should migrate to the datalad-osf extension. however, right\n    # now extensions cannot provide default configuration\n    # https://github.com/datalad/datalad/issues/5769\n    'datalad.clone.url-substitute.osf': {\n        'ui': ('question', {\n               'title': 'Open Science Framework URL substitution rule',\n               'text': 'Mangling for OSF-related URLs. ' + subst_rule_docs\n        }),\n        'destination': 'global',\n        'default': (\n            # accept browser-provided URL and convert to those accepted by\n            # the datalad-osf extension\n            r',^https://osf.io/([^/]+)[/]*$,osf://\\1',\n        )\n    },\n    # this is actually used in downloaders, but kept cfg name original\n    'datalad.crawl.cache': {\n        'ui': ('yesno', {\n               'title': 'Crawler download caching',\n               'text': 'Should the crawler cache downloaded files?'}),\n        'destination': 'local',\n        'type': bool,\n    },\n    # this is actually used in downloaders, but kept cfg name original\n    'datalad.credentials.force-ask': {\n        'ui': ('yesno', {\n               'title': 'Force (re-)entry of credentials',\n               'text': 'Should DataLad prompt for credential (re-)entry? This '\n                       'can be used to update previously stored credentials.'}),\n        'type': bool,\n        'default': False,\n    },\n    'datalad.externals.nda.dbserver': {\n        'ui': ('question', {\n               'title': 'NDA database server',\n               'text': 'Hostname of the database server'}),\n        'destination': 'global',\n        # Development one is https://development.nimhda.org\n        'default': 'https://nda.nih.gov/DataManager/dataManager',\n    },\n    'datalad.locations.cache': {\n        'ui': ('question', {\n               'title': 'Cache directory',\n               'text': 'Where should datalad cache files?'}),\n        'destination': 'global',\n        'default_fn': lambda: dirs.user_cache_dir,\n    },\n    'datalad.locations.default-dataset': {\n        'ui': ('question', {\n               'title': 'Default dataset path',\n               'text': 'Where should datalad should look for (or install) a '\n                       'default dataset?'}),\n        'destination': 'global',\n        'default_fn': lambda: opj(expanduser('~'), 'datalad'),\n    },\n    'datalad.locations.locks': {\n        'ui': ('question', {\n               'title': 'Lockfile directory',\n               'text': 'Where should datalad store lock files?'}),\n        'destination': 'global',\n        'default_fn': lambda: opj(dirs.user_cache_dir, 'locks')\n    },\n    'datalad.locations.sockets': {\n        'ui': ('question', {\n               'title': 'Socket directory',\n               'text': 'Where should datalad store socket files?'}),\n        'destination': 'global',\n        'default_fn': lambda: opj(dirs.user_cache_dir, 'sockets'),\n    },\n    'datalad.locations.system-procedures': {\n        'ui': ('question', {\n               'title': 'System procedure directory',\n               'text': 'Where should datalad search for system procedures?'}),\n        'destination': 'global',\n        'default_fn': lambda: opj(dirs.site_config_dir, 'procedures'),\n    },\n    'datalad.locations.user-procedures': {\n        'ui': ('question', {\n               'title': 'User procedure directory',\n               'text': 'Where should datalad search for user procedures?'}),\n        'destination': 'global',\n        'default_fn': lambda: opj(dirs.user_config_dir, 'procedures'),\n    },\n    'datalad.locations.extra-procedures': {\n        'ui': ('question', {\n            'title': 'Extra procedure directory',\n            'text': 'Where should datalad search for some additional procedures?'}),\n        'destination': 'global',\n    },\n    'datalad.locations.dataset-procedures': {\n        'ui': ('question', {\n               'title': 'Dataset procedure directory',\n               'text': 'Where should datalad search for dataset procedures (relative to a dataset root)?'}),\n        'destination': 'dataset',\n        'default': opj('.datalad', 'procedures'),\n    },\n    'datalad.exc.str.tblimit': {\n        'ui': ('question', {\n               'title': 'This flag is used by datalad to cap the number of traceback steps included in exception logging and result reporting to DATALAD_EXC_STR_TBLIMIT of pre-processed entries from traceback.'}),\n    },\n    'datalad.fake-dates': {\n        'ui': ('yesno', {\n               'title': 'Fake (anonymize) dates',\n               'text': 'Should the dates in the logs be faked?'}),\n        'destination': 'local',\n        'type': EnsureBool(),\n        'default': False,\n    },\n    'datalad.fake-dates-start': {\n        'ui': ('question', {\n            'title': 'Initial fake date',\n            'text': 'When faking dates and there are no commits in any local branches, generate the date by adding one second to this value (Unix epoch time). The value must be positive.'}),\n        'type': EnsureInt(),\n        'default': 1112911993,\n    },\n    'datalad.github.token-note': {\n        'ui': ('question', {\n            'title': 'Github token note',\n            'text': 'Description for a Personal access token to generate.'}),\n        'default': 'DataLad',\n    },\n    'datalad.tests.nonetwork': {\n        'ui': ('yesno', {\n               'title': 'Skips network tests completely if this flag is set Examples include test for s3, git_repositories, openfmri etc'}),\n        'type': EnsureBool(),\n    },\n    'datalad.tests.nonlo': {\n        'ui': ('question', {\n               'title': 'Specifies network interfaces to bring down/up for testing. Currently used by travis.'}),\n    },\n    'datalad.tests.noteardown': {\n        'ui': ('yesno', {\n               'title': 'Does not execute teardown_package which cleans up temp files and directories created by tests if this flag is set'}),\n        'type': EnsureBool(),\n    },\n    'datalad.tests.dataladremote': {\n        'ui': ('yesno', {\n               'title': 'Binary flag to specify whether each annex repository should get datalad special remote in every test repository'}),\n        'type': EnsureBool(),\n    },\n    'datalad.tests.runcmdline': {\n        'ui': ('yesno', {\n               'title': 'Binary flag to specify if shell testing using shunit2 to be carried out'}),\n        'type': EnsureBool(),\n    },\n    'datalad.tests.ssh': {\n        'ui': ('yesno', {\n               'title': 'Skips SSH tests if this flag is **not** set'}),\n        'type': EnsureBool(),\n    },\n    'datalad.tests.knownfailures.skip': {\n        'ui': ('yesno', {\n               'title': 'Skips tests that are known to currently fail'}),\n        'type': EnsureBool(),\n        'default': True,\n    },\n    'datalad.tests.knownfailures.probe': {\n        'ui': ('yesno', {\n               'title': 'Probes tests that are known to fail on whether or not they are actually still failing'}),\n        'type': EnsureBool(),\n        'default': False,\n    },\n    'datalad.tests.setup.testrepos': {\n        'ui': ('question', {\n            'title': 'Pre-creates repositories for @with_testrepos within setup_package'}),\n        'type': EnsureBool(),\n        'default': False,\n    },\n    'datalad.tests.temp.dir': {\n        'ui': ('question', {\n               'title': 'Create a temporary directory at location specified by this flag. It is used by tests to create a temporary git directory while testing git annex archives etc'}),\n        'type': EnsureStr(),\n        'default_fn': lambda: environ.get('TMPDIR'),\n    },\n    'datalad.tests.temp.keep': {\n        'ui': ('yesno', {\n               'title': 'Function rmtemp will not remove temporary file/directory created for testing if this flag is set'}),\n        'type': EnsureBool(),\n    },\n    'datalad.tests.temp.fs': {\n        'ui': ('question', {\n               'title': 'Specify the temporary file system to use as loop device for testing DATALAD_TESTS_TEMP_DIR creation'}),\n    },\n    'datalad.tests.temp.fssize': {\n        'ui': ('question', {\n               'title': 'Specify the size of temporary file system to use as loop device for testing DATALAD_TESTS_TEMP_DIR creation'}),\n    },\n    'datalad.tests.ui.backend': {\n        'ui': ('question', {\n            'title': 'Tests UI backend',\n            # XXX we could add choices...\n            'text': 'Which UI backend to use'}),\n        'default': 'tests-noninteractive',\n    },\n    'datalad.tests.usecassette': {\n        'ui': ('question', {\n               'title': 'Specifies the location of the file to record network transactions by the VCR module. Currently used by when testing custom special remotes'}),\n    },\n    'datalad.tests.cache': {\n        'ui': ('question', {\n            'title': 'Cache directory for tests',\n            'text': 'Where should datalad cache test files?'}),\n        'destination': 'global',\n        'default_fn': lambda: opj(dirs.user_cache_dir, 'tests')\n    },\n    'datalad.log.level': {\n        'ui': ('question', {\n            'title': 'Used for control the verbosity of logs printed to '\n                     'stdout while running datalad commands/debugging'}),\n    },\n    'datalad.log.result-level': {\n        'ui': ('question', {\n               'title': 'Log level for command result messages',\n               'text': \"Overrides the default behavior of logging 'impossible' \"\n                       \"results as a warning, 'error' results as errors, and \"\n                       \"everything else as 'debug' with a single alternative \"\n                       \"log level\"}),\n        'type': EnsureChoice('debug', 'info', 'warning', 'error'),\n        # None keeps the default behavior\n        'default': 'None',\n    },\n    'datalad.log.name': {\n        'ui': ('question', {\n            'title': 'Include name of the log target in the log line'}),\n    },\n    'datalad.log.names': {\n        'ui': ('question', {\n            'title': 'Which names (,-separated) to print log lines for'}),\n    },\n    'datalad.log.namesre': {\n        'ui': ('question', {\n            'title': 'Regular expression for which names to print log lines for'}),\n    },\n    'datalad.log.outputs': {\n        'ui': ('question', {\n               'title': 'Whether to log stdout and stderr for executed commands',\n               'text': 'When enabled, setting the log level to 5 '\n                       'should catch all execution output, '\n                       'though some output may be logged at higher levels'}),\n        'default': False,\n        'type': EnsureBool(),\n    },\n    'datalad.log.timestamp': {\n        'ui': ('yesno', {\n               'title': 'Used to add timestamp to datalad logs'}),\n        'default': False,\n        'type': EnsureBool(),\n    },\n    'datalad.log.traceback': {\n        'ui': ('question', {\n               'title': 'Runs TraceBack function with collide set to True, if this flag is set to \"collide\". This replaces any common prefix between current traceback log and previous invocation with \"...\"'}),\n    },\n    'datalad.log.exc': {\n        'ui': ('yesno', {\n               'title': 'Include exceptions and their traceback in log messages. If set, \\'datalad.exc.str.tblimit\\' applies.'}),\n        'default': False,\n        'type': EnsureBool(),\n    },\n    'datalad.ssh.identityfile': {\n        'ui': ('question', {\n               'title': \"If set, pass this file as ssh's -i option.\"}),\n        'destination': 'global',\n        'default': None,\n    },\n    'datalad.ssh.multiplex-connections': {\n        'ui': ('question', {\n               'title': \"Whether to use a single shared connection for multiple SSH processes aiming at the same target.\"}),\n        'destination': 'global',\n        'default': not on_windows,\n        'type': EnsureBool(),\n    },\n    'datalad.annex.retry': {\n        'ui': ('question',\n               {'title': 'Value for annex.retry to use for git-annex calls',\n                'text': 'On transfer failure, annex.retry (sans \"datalad.\") '\n                        'controls the number of times that git-annex retries. '\n                        'DataLad will call git-annex with annex.retry set '\n                        'to the value here unless the annex.retry '\n                        'is explicitly configured'}),\n        'type': EnsureInt(),\n        'default': 3,\n    },\n    'datalad.repo.backend': {\n        'ui': ('question', {\n               'title': 'git-annex backend',\n               'text': 'Backend to use when creating git-annex repositories'}),\n        'default': 'MD5E',\n    },\n    'datalad.repo.direct': {\n        'ui': ('yesno', {\n               'title': 'Direct Mode for git-annex repositories',\n               'text': 'Set this flag to create annex repositories in direct mode by default'}),\n        'type': EnsureBool(),\n        'default': False,\n    },\n    'datalad.repo.version': {\n        'ui': ('question', {\n               'title': 'git-annex repository version',\n               'text': 'Specifies the repository version for git-annex to be used by default'}),\n        'type': EnsureInt(),\n        'default': 8,\n    },\n    'datalad.metadata.maxfieldsize': {\n        'ui': ('question', {\n               'title': 'Maximum metadata field size',\n               'text': 'Metadata fields exceeding this size (in bytes/chars) are excluded from metadata extractio'}),\n        'default': 100000,\n        'type': EnsureInt(),\n    },\n    'datalad.metadata.nativetype': {\n        'ui': ('question', {\n               'title': 'Native dataset metadata scheme',\n               'text': 'Set this label to engage a particular metadata extraction parser'}),\n    },\n    'datalad.metadata.store-aggregate-content': {\n        'ui': ('question', {\n               'title': 'Aggregated content metadata storage',\n               'text': 'If this flag is enabled, content metadata is aggregated into superdataset to allow for discovery of individual files. If disable unique content metadata values are still aggregated to enable dataset discovery'}),\n        'type': EnsureBool(),\n        'default': True,\n    },\n    'datalad.search.default-mode': {\n        'ui': ('question', {\n               'title': 'Default search mode',\n               'text': 'Label of the mode to be used by default'}),\n        'type': EnsureChoice('egrep', 'textblob', 'autofield'),  # graph,...\n        'default': 'egrep',\n    },\n    'datalad.search.index-default-documenttype': {\n        'ui': ('question', {\n               'title': 'Type of search index documents',\n               'text': 'Labels of document types to include in a default search index'}),\n        'type': EnsureChoice('all', 'datasets', 'files'),\n        'default': 'datasets',\n    },\n    'datalad.metadata.create-aggregate-annex-limit': {\n        'ui': ('question', {\n               'title': 'Limit configuration annexing aggregated metadata in new dataset',\n               'text': 'Git-annex large files expression (see https://git-annex.branchable.com/tips/largefiles; given expression will be wrapped in parentheses)'}),\n        'default': 'anything',\n    },\n    'datalad.runtime.max-annex-jobs': {\n        'ui': ('question', {\n               'title': 'Maximum number of git-annex jobs to request when \"jobs\" option set to \"auto\" (default)',\n               'text': 'Set this value to enable parallel annex jobs that may speed up certain operations (e.g. get file content). The effective number of jobs will not exceed the number of available CPU cores (or 3 if there is less than 3 cores).'}),\n        'type': EnsureInt(),\n        'default': 1,\n    },\n    'datalad.runtime.max-jobs': {\n        'ui': ('question', {\n            'title': 'Maximum number of jobs DataLad can run in \"parallel\"',\n            'text': 'Set this value to enable parallel multi-threaded DataLad jobs that may speed up certain '\n                    'operations, in particular operation across multiple datasets (e.g., install multiple '\n                    'subdatasets, etc).'}),\n        'type': EnsureInt(),\n        'default': 1,\n    },\n    'datalad.runtime.raiseonerror': {\n        'ui': ('question', {\n               'title': 'Error behavior',\n               'text': 'Set this flag to cause DataLad to raise an exception on errors that would have otherwise just get logged'}),\n        'type': EnsureBool(),\n        'default': False,\n    },\n    'datalad.runtime.report-status': {\n        'ui': ('question', {\n               'title': 'Command line result reporting behavior',\n               'text': \"If set (to other than 'all'), constrains command result report to records matching the given status. 'success' is a synonym for 'ok' OR 'notneeded', 'failure' stands for 'impossible' OR 'error'\"}),\n        'type': EnsureChoice('all', 'success', 'failure', 'ok', 'notneeded', 'impossible', 'error'),\n        'default': None,\n    },\n    'datalad.runtime.stalled-external': {\n        'ui': ('question', {\n            'title': 'Behavior for handing external processes',\n            'text': 'What to do with external processes if they do not finish in some minimal reasonable time. '\n                    'If \"abandon\", datalad would proceed without waiting for external process to exit. '\n                    'ATM applies only to batched git-annex processes. Should be changed with caution.'}),\n        'type': EnsureChoice('wait', 'abandon'),\n        'default': 'wait',\n    },\n    'datalad.search.indexercachesize': {\n        'ui': ('question', {\n               'title': 'Maximum cache size for search index (per process)',\n               'text': 'Actual memory consumption can be twice as high as this value in MB (one process per CPU is used)'}),\n        'default': 256,\n        'type': EnsureInt(),\n    },\n    'datalad.ui.progressbar': {\n        'ui': ('question', {\n            'title': 'UI progress bars',\n            'text': 'Default backend for progress reporting'}),\n        'default': None,\n        'type': EnsureChoice('tqdm', 'tqdm-ipython', 'log', 'none'),\n    },\n    'datalad.ui.color': {\n        'ui': ('question', {\n            'title': 'Colored terminal output',\n            'text': 'Enable or disable ANSI color codes in outputs; \"on\" overrides NO_COLOR environment variable'}),\n        'default': 'auto',\n        'type': EnsureChoice('on', 'off', 'auto'),\n    },\n    'datalad.ui.suppress-similar-results': {\n        'ui': ('question', {\n            'title': 'Suppress rendering of similar repetitive results',\n            'text': \"If enabled, after a certain number of subsequent \"\n                    \"results that are identical regarding key properties, \"\n                    \"such as 'status', 'action', and 'type', additional \"\n                    \"similar results are not rendered by the common result \"\n                    \"renderer anymore. Instead, a count \"\n                    \"of suppressed results is displayed. If disabled, or \"\n                    \"when not running in an interactive terminal, all results \"\n                    \"are rendered.\"}),\n        'default': True,\n        'type': EnsureBool(),\n    },\n    'datalad.ui.suppress-similar-results-threshold': {\n        'ui': ('question', {\n            'title': 'Threshold for suppressing similar repetitive results',\n            'text': \"Minimum number of similar results to occur before \"\n                    \"suppression is considered. \"\n                    \"See 'datalad.ui.suppress-similar-results' for more \"\n                    \"information.\"}),\n        'default': 10,\n        'type': EnsureInt(),\n    },\n    'datalad.save.no-message': {\n        'ui': ('question', {\n            'title': 'Commit message handling',\n            'text': 'When no commit message was provided: '\n                    'attempt to obtain one interactively (interactive); '\n                    'or use a generic commit message (generic). '\n                    'NOTE: The interactive option is experimental. The '\n                    'behavior may change in backwards-incompatible ways.'}),\n        'default': 'generic',\n        'type': EnsureChoice('interactive', 'generic'),\n    },\n    'datalad.install.inherit-local-origin': {\n        'ui': ('question', {\n            'title': 'Inherit local origin of dataset source',\n            'text': \"If enabled, a local 'origin' remote of a local dataset \"\n                    \"clone source is configured as an 'origin-2' remote \"\n                    \"to make its annex automatically available. The process \"\n                    \"is repeated recursively for any further qualifying \"\n                    \"'origin' dataset thereof.\"\n                    \"Note that if clone.defaultRemoteName is configured \"\n                    \"to use a name other than 'origin', that name will be \"\n                    \"used instead.\"}),\n        'default': True,\n        'type': EnsureBool(),\n    },\n}",
                            "ast_type_str": "Assign",
                            "start_lineno": 41,
                            "end_lineno": 510,
                            "parent": null
                        }
                    ],
                    "deleted_entities": []
                }
            ]
        }
    ],
    "old_commit_hash": "70c8056eeaadd26c1ff547edf520f274b6562356^",
    "new_commit_hash": "70c8056eeaadd26c1ff547edf520f274b6562356",
    "commit_message": "NF: Support credential updates\n\nThis change introduces a new configuration setting:\n\n```\n % datalad x-configuration\n ...\n # Force (re-)entry of credentials\n # Should DataLad prompt for credential (re-)entry? This can be used to\n # update previously stored credentials.\n datalad.credentials.force-ask=False\n ...\n```\n\nIt enables seamless updates of stored credentials. Importantly, users\nare no longer required to learn where credentials are stored, and how to\nupdate them in that store. It makes instructions provided in the\nhandbook on how to learn python `keyring` superfluous. Credential\nupdates are simply entered at the same place where the original\ncredentials were provided.\n\nThis is not a perfect solution (can get messy when there are a\nnumber of credentials involved in a single datalad operation, and only a\nsingle one needs updating), but it addresses a substantial usability\nissue for a broad range of use cases in a straightforward manner.\n\nBesides being compatible with the usual ways of DataLad configuration\nhandling, this also works for plain git-annex calls that trigger\nDataLad's credential handling, e.g.:\n\n```\ngit -c datalad.credentials.force-ask=no annex get T1w_acpc_dc.nii.gz\n```\n\nFixes datalad/datalad#5775",
    "commit_date": "2021-07-16T10:10:00+02:00",
    "metadata": {}
}
{
    "repo_name": "datalad",
    "new_commit_hash": "70c8056eeaadd26c1ff547edf520f274b6562356",
    "test_file_codes": [
        "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Tests for credentials\"\"\"\n\nfrom unittest.mock import patch\nfrom datalad.tests.utils import (\n    assert_equal,\n    assert_false,\n    assert_in,\n    assert_raises,\n    assert_true,\n    SkipTest,\n    with_testsui,\n)\nfrom datalad.support.keyring_ import (\n    Keyring,\n    MemoryKeyring,\n)\nfrom datalad.downloaders.credentials import (\n    AWS_S3,\n    CompositeCredential,\n    UserPassword,\n)\nfrom datalad import cfg as dlcfg\n\n\n@with_testsui(responses=[\n    'user1', 'password1',\n    # when we do provide user to enter_new\n    'newpassword',\n])\ndef test_cred1_enter_new():\n    keyring = MemoryKeyring()\n    cred = UserPassword(\"name\", keyring=keyring)\n    assert_false(cred.is_known)\n    assert_equal(cred.enter_new(), None)\n    assert_true(cred.is_known)\n    assert_equal(keyring.get('name', 'user'), 'user1')\n    assert_equal(keyring.get('name', 'password'), 'password1')\n    keyring.delete('name')\n    assert_raises(KeyError, keyring.delete, 'name', 'user')\n    assert_raises(KeyError, keyring.delete, 'name')\n    assert_equal(keyring.get('name', 'user'), None)\n\n    # Test it blowing up if we provide unknown field\n    with assert_raises(ValueError) as cme:\n        cred.enter_new(username='user')\n    assert_in('field(s): username.  Known but not specified: password, user',\n              str(cme.exception))\n\n    # Test that if user is provided, it is not asked\n    cred.enter_new(user='user2')\n    assert_equal(keyring.get('name', 'user'), 'user2')\n    assert_equal(keyring.get('name', 'password'), 'newpassword')\n\n@with_testsui(responses=['password1', 'newuser', 'newpassword'])\ndef test_cred1_call():\n    keyring = MemoryKeyring()\n    cred = UserPassword(\"name\", keyring=keyring)\n    # we will set the name but not the password, expecting UI\n    # requesting it\n    assert_equal(keyring.set('name', 'user', 'user1'), None)\n    assert_equal(keyring.get('name', 'user'), 'user1')\n    assert_equal(cred(), {'user': 'user1', 'password': 'password1'})\n    assert_equal(keyring.get('name', 'password'), 'password1')\n    # without intervention the same credentials will be reused\n    # in subsequent attempts\n    assert_equal(cred(), {'user': 'user1', 'password': 'password1'})\n    with patch.dict(dlcfg._merged_store, {'datalad.credentials.force-ask': 'yes'}):\n        assert_equal(cred(), {'user': 'newuser', 'password': 'newpassword'})\n    assert_equal(keyring.get('name', 'user'), 'newuser')\n    assert_equal(keyring.get('name', 'password'), 'newpassword')\n\n\ndef test_keyring():\n    # mock out keyring methods and test that we are providing correct values\n    # with 'datalad-' prefix\n    raise SkipTest(\"provide tests for Keyring which interfaces keyring module\")\n\n\ndef _cred1_adapter(composite, user=None, password=None):\n    \"\"\"Just a sample adapter from one user/pw type to another\"\"\"\n    return dict(user=user + \"_1\", password=password + \"_2\")\n\n\nclass _CCred1(CompositeCredential):\n    \"\"\"A Simple composite credential which will do some entries transformation\n    \"\"\"\n    _CREDENTIAL_CLASSES = (UserPassword, UserPassword)\n    _CREDENTIAL_ADAPTERS = (_cred1_adapter,)\n\n\n@with_testsui(responses=['user1', 'password1',\n                         'user2', 'password2'])\ndef test_composite_credential1():\n    # basic test of composite credential\n    keyring = MemoryKeyring()\n    cred = _CCred1(\"name\", keyring=keyring)\n    # When queried, does the chain\n    assert_equal(cred(), {'user': 'user1_1', 'password': 'password1_2'})\n    # But the \"Front\" credential is exposed to the user\n    assert_equal(cred.get('user'), 'user1')\n    assert_equal(keyring.get('name', 'user'), 'user1')\n    assert_raises(ValueError, cred.get, 'unknown_field')\n    assert_equal(cred.get('password'), 'password1')\n    assert_equal(keyring.get('name', 'password'), 'password1')\n    # ATM composite credential stores \"derived\" ones unconditionally in the\n    # keyring as well\n    assert_equal(keyring.get('name:1', 'user'), 'user1_1')\n    assert_equal(keyring.get('name:1', 'password'), 'password1_2')\n\n    # and now enter new should remove \"derived\" entries\n    cred.enter_new()\n    assert_equal(keyring.get('name', 'user'), 'user2')\n    assert_equal(keyring.get('name', 'password'), 'password2')\n    # we immediately refresh all credentials in the chain\n    assert_equal(keyring.get('name:1', 'user'), 'user2_1')\n    assert_equal(keyring.get('name:1', 'password'), 'password2_2')\n    assert_equal(cred(), {'user': 'user2_1', 'password': 'password2_2'})\n\n\ndef test_credentials_from_env():\n    keyring = Keyring()\n    cred = AWS_S3(\"test-s3\", keyring=keyring)\n    assert_false(cred.is_known)\n    assert_equal(cred.get('key_id'), None)\n    assert_equal(cred.get('secret_id'), None)\n\n    def _check1():\n        assert_equal(cred.get('key_id'), '1')\n        assert_false(cred.is_known)\n\n    def _check2():\n        assert_equal(cred.get('key_id'), '1')\n        assert_equal(cred.get('secret_id'), '2')\n        assert_true(cred.is_known)\n\n    # this is the old way, should still work\n    with patch.dict('os.environ', {'DATALAD_test_s3_key_id': '1'}):\n        _check1()\n        with patch.dict('os.environ', {'DATALAD_test_s3_secret_id': '2'}):\n            _check2()\n        assert_false(cred.is_known)  # no memory of the past\n\n    # here is the new way\n    import datalad\n    try:\n        with patch.dict('os.environ', {'DATALAD_CREDENTIAL_test__s3_key__id': '1'}):\n            datalad.cfg.reload()\n            _check1()\n            with patch.dict('os.environ', {'DATALAD_CREDENTIAL_test__s3_secret__id': '2'}):\n                datalad.cfg.reload()\n                _check2()\n            datalad.cfg.reload()\n            assert_false(cred.is_known)  # no memory of the past\n    finally:\n        datalad.cfg.reload()\n",
        "import pytest\nimport tempfile\nimport os\n\n\n@pytest.fixture\ndef path():\n    with tempfile.TemporaryDirectory() as tmpdir:\n        yield tmpdir\n\n\n@pytest.fixture(autouse=True)\ndef setup_git_config():\n    with tempfile.TemporaryDirectory() as home:\n        old_home = os.environ.get(\"HOME\")\n        os.environ[\"HOME\"] = home\n        os.system('git config --global user.name \"DataLad Tester\"')\n        os.system('git config --global user.email \"test@example.com\"')\n        yield\n        if old_home:\n            os.environ[\"HOME\"] = old_home\n"
    ],
    "test_file_names": [
        "test_1.py",
        "conftest.py"
    ],
    "setup_res_code": 0,
    "setup_res_stdout": "Starting Datalad installation attempts...\nVerifying installation...\nDatalad version: 0.14.4.dev462\n✅ Installation successful!\nSuccessfully installed Datalad using Python 3.9\n",
    "setup_res_stderr": "Using CPython 3.9.21\nCreating virtual environment at: .venv\nActivate with: source .venv/bin/activate\nResolved 25 packages in 105ms\nInstalled 25 packages in 2.54s\n + coverage==7.6.10\n + docopt==0.6.2\n + exceptiongroup==1.2.2\n + grabbit==0.2.6\n + importlib-resources==6.4.5\n + iniconfig==2.0.0\n + nibabel==5.3.2\n + num2words==0.5.14\n + numpy==2.0.2\n + packaging==24.2\n + pandas==2.2.3\n + patsy==1.0.1\n + pluggy==1.5.0\n + pybids==0.6.5\n + pytest==8.3.4\n + pytest-cov==6.0.0\n + python-dateutil==2.9.0.post0\n + pytz==2024.2\n + scipy==1.13.1\n + setuptools==75.6.0\n + six==1.17.0\n + tomli==2.2.1\n + typing-extensions==4.12.2\n + tzdata==2024.2\n + zipp==3.21.0\nResolved 62 packages in 3.93s\nPrepared 1 package in 3.00s\nInstalled 56 packages in 227ms\n + annexremote==1.6.6\n + appdirs==1.4.4\n + backports-tarfile==1.2.0\n + beautifulsoup4==4.12.3\n + boto==2.49.0\n + certifi==2024.12.14\n + cffi==1.17.1\n + chardet==5.2.0\n + charset-normalizer==3.4.1\n + citeproc-py==0.6.0\n + cryptography==44.0.0\n + datalad==0.14.4 (from file:///home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_70c8056eeaadd26c1ff547edf520f274b6562356)\n + deprecated==1.2.15\n + distro==1.9.0\n + duecredit==0.10.2\n + exifread==3.0.0\n + fasteners==0.19\n + httpretty==1.1.4\n + humanize==4.11.0\n + idna==3.10\n + importlib-metadata==8.5.0\n + iso8601==2.1.0\n + jaraco-classes==3.4.0\n + jaraco-context==6.0.1\n + jaraco-functools==4.1.0\n + jeepney==0.8.0\n + keyring==25.6.0\n + keyrings-alt==5.0.2\n + looseversion==1.3.0\n + lxml==5.3.0\n + more-itertools==10.5.0\n + msgpack==1.1.0\n + multidict==6.1.0\n + mutagen==1.47.0\n + nose==1.3.7\n + patool==1.15.0\n + pillow==11.0.0\n + propcache==0.2.1\n + pycparser==2.22\n + pygithub==2.5.0\n + pyjwt==2.10.1\n + pynacl==1.5.0\n + pyperclip==1.9.0\n + python-xmp-toolkit==2.0.2\n + pyyaml==6.0.2\n + requests==2.32.3\n + requests-ftp==0.3.1\n + secretstorage==3.3.3\n + simplejson==3.19.3\n + soupsieve==2.6\n + tqdm==4.67.1\n + urllib3==1.26.20\n + vcrpy==7.0.0\n + whoosh==2.7.4\n + wrapt==1.17.0\n + yarl==1.18.3\nResolved 96 packages in 347ms\nInstalled 32 packages in 1.19s\n + alabaster==0.7.16\n + asv==0.6.4\n + asv-runner==0.2.1\n + babel==2.16.0\n + build==1.2.2.post1\n + distlib==0.3.9\n + docutils==0.21.2\n + filelock==3.16.1\n + gprof2dot==2024.6.6\n + imagesize==1.4.1\n + jinja2==3.1.5\n + json5==0.10.0\n + markupsafe==3.0.2\n + nose-timer==1.0.1\n + platformdirs==4.3.6\n + psutil==6.1.1\n + pygments==2.18.0\n + pympler==1.1\n + pypandoc==1.14\n + pyproject-hooks==1.2.0\n + snowballstemmer==2.2.0\n + sphinx==7.4.7\n + sphinx-rtd-theme==3.0.2\n + sphinxcontrib-applehelp==2.0.0\n + sphinxcontrib-devhelp==2.0.0\n + sphinxcontrib-htmlhelp==2.1.0\n + sphinxcontrib-jquery==4.1\n + sphinxcontrib-jsmath==1.0.1\n + sphinxcontrib-qthelp==2.0.0\n + sphinxcontrib-serializinghtml==2.0.0\n + tabulate==0.9.0\n + virtualenv==20.28.0\n",
    "new_commit_res_code": 0,
    "new_commit_res_stdout": "============================= test session starts ==============================\nplatform linux -- Python 3.9.21, pytest-8.3.4, pluggy-1.5.0\nrootdir: /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_70c8056eeaadd26c1ff547edf520f274b6562356\nconfigfile: pyproject.toml\nplugins: cov-6.0.0\ncollected 5 items\n\nr2e_tests/test_1.py ..s..                                                [100%]\n\n==================================== PASSES ====================================\n=========================== short test summary info ============================\nPASSED r2e_tests/test_1.py::test_cred1_enter_new\nPASSED r2e_tests/test_1.py::test_cred1_call\nPASSED r2e_tests/test_1.py::test_composite_credential1\nPASSED r2e_tests/test_1.py::test_credentials_from_env\nSKIPPED [1] .venv/lib/python3.9/site-packages/_pytest/unittest.py:385: provide tests for Keyring which interfaces keyring module\n========================= 4 passed, 1 skipped in 4.87s =========================\n",
    "new_commit_res_stderr": "",
    "old_commit_res_code": 1,
    "old_commit_res_stdout": "============================= test session starts ==============================\nplatform linux -- Python 3.9.21, pytest-8.3.4, pluggy-1.5.0\nrootdir: /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_70c8056eeaadd26c1ff547edf520f274b6562356\nconfigfile: pyproject.toml\nplugins: cov-6.0.0\ncollected 5 items\n\nr2e_tests/test_1.py .Fs..                                                [100%]\n\n=================================== FAILURES ===================================\n_______________________________ test_cred1_call ________________________________\n\n    @with_testsui(responses=['password1', 'newuser', 'newpassword'])\n    def test_cred1_call():\n        keyring = MemoryKeyring()\n        cred = UserPassword(\"name\", keyring=keyring)\n        # we will set the name but not the password, expecting UI\n        # requesting it\n        assert_equal(keyring.set('name', 'user', 'user1'), None)\n        assert_equal(keyring.get('name', 'user'), 'user1')\n        assert_equal(cred(), {'user': 'user1', 'password': 'password1'})\n        assert_equal(keyring.get('name', 'password'), 'password1')\n        # without intervention the same credentials will be reused\n        # in subsequent attempts\n        assert_equal(cred(), {'user': 'user1', 'password': 'password1'})\n        with patch.dict(dlcfg._merged_store, {'datalad.credentials.force-ask': 'yes'}):\n>           assert_equal(cred(), {'user': 'newuser', 'password': 'newpassword'})\n\nr2e_tests/test_1.py:76: \n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \n../../../../.local/share/uv/python/cpython-3.9.21-linux-x86_64-gnu/lib/python3.9/unittest/case.py:837: in assertEqual\n    assertion_func(first, second, msg=msg)\n../../../../.local/share/uv/python/cpython-3.9.21-linux-x86_64-gnu/lib/python3.9/unittest/case.py:1136: in assertDictEqual\n    self.fail(self._formatMessage(msg, standardMsg))\n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \n\nself = <nose.tools.trivial.Dummy testMethod=nop>\nmsg = \"{'user': 'user1', 'password': 'password1'} != {'user': 'newuser', 'password': 'newpassword'}\\n- {'password': 'passwor...  -                -\\n\\n+ {'password': 'newpassword', 'user': 'newuser'}\\n?               +++                    +++\\n\"\n\n    def fail(self, msg=None):\n        \"\"\"Fail immediately, with the given message.\"\"\"\n>       raise self.failureException(msg)\nE       AssertionError: {'user': 'user1', 'password': 'password1'} != {'user': 'newuser', 'password': 'newpassword'}\nE       - {'password': 'password1', 'user': 'user1'}\nE       ?                       -                -\nE       \nE       + {'password': 'newpassword', 'user': 'newuser'}\nE       ?               +++                    +++\n\n../../../../.local/share/uv/python/cpython-3.9.21-linux-x86_64-gnu/lib/python3.9/unittest/case.py:676: AssertionError\n==================================== PASSES ====================================\n=========================== short test summary info ============================\nPASSED r2e_tests/test_1.py::test_cred1_enter_new\nPASSED r2e_tests/test_1.py::test_composite_credential1\nPASSED r2e_tests/test_1.py::test_credentials_from_env\nSKIPPED [1] .venv/lib/python3.9/site-packages/_pytest/unittest.py:385: provide tests for Keyring which interfaces keyring module\nFAILED r2e_tests/test_1.py::test_cred1_call - AssertionError: {'user': 'user1...\n==================== 1 failed, 3 passed, 1 skipped in 0.71s ====================\n",
    "old_commit_res_stderr": ""
}
datalad/downloaders/credentials.py datalad/downloaders/tests/test_credentials.py datalad/interface/common_cfg.py
We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
[ISSUE]

**Title:** Forced Credential Re-entry Not Triggered with `datalad.credentials.force-ask` Configuration

**Description:**

When attempting to update stored credentials by enabling the `datalad.credentials.force-ask` configuration setting, DataLad does not prompt for credential re-entry. This prevents users from updating their credentials seamlessly, as the existing credentials remain unchanged.

**Example Code:**

```python
from datalad.downloaders.credentials import UserPassword
from datalad import cfg as dlcfg

# Initialize keyring and set initial credentials
keyring = MemoryKeyring()
cred = UserPassword("name", keyring=keyring)
keyring.set('name', 'user', 'user1')
keyring.set('name', 'password', 'password1')

# Retrieve existing credentials
credentials = cred()
print(credentials)  # Outputs: {'user': 'user1', 'password': 'password1'}

# Attempt to force credential re-entry
dlcfg.set('datalad.credentials.force-ask', True)
credentials = cred()
print(credentials)  # Expected to prompt for new credentials and update
```

**Expected Behavior:**

When `datalad.credentials.force-ask` is set to `True`, calling `cred()` should prompt the user to re-enter their credentials. The new credentials provided by the user should then replace the existing ones in the keyring.

**Actual Behavior:**

Despite setting `force-ask` to `True`, calling `cred()` does not prompt for re-entry. The original credentials `{'user': 'user1', 'password': 'password1'}` are returned without any update, making it impossible to update the stored credentials through this configuration.

[/ISSUE]
--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.

--- BEGIN FILE ---
```
### datalad/interface/common_cfg.py
# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-
# ex: set sts=4 ts=4 sw=4 noet:
# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
#
#   See COPYING file distributed along with the datalad package for the
#   copyright and license terms.
#
# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
"""Common configuration options

"""

__docformat__ = 'restructuredtext'

from appdirs import AppDirs
from os import environ
from os.path import join as opj, expanduser
from datalad.support.constraints import EnsureBool
from datalad.support.constraints import EnsureInt
from datalad.support.constraints import EnsureNone
from datalad.support.constraints import EnsureChoice
from datalad.support.constraints import EnsureListOf
from datalad.support.constraints import EnsureStr
from datalad.utils import on_windows

dirs = AppDirs("datalad", "datalad.org")

subst_rule_docs = """\
A substitution specification is a string with a match and substitution
expression, each following Python's regular expression syntax. Both expressions
are concatenated to a single string with an arbitrary delimiter character. The
delimiter is defined by prefixing the string with the delimiter. Prefix and
delimiter are stripped from the expressions (Example:
",^http://(.*)$,https://\\1").  This setting can be defined multiple times.
Substitutions will be applied incrementally, in order of their definition. The
first substitution in such a series must match, otherwise no further
substitutions in a series will be considered. However, following the first
match all further substitutions in a series are processed, regardless whether
intermediate expressions match or not."""

definitions = {
    'datalad.clone.url-substitute.github': {
        'ui': ('question', {
               'title': 'GitHub URL substitution rule',
               'text': 'Mangling for GitHub-related URL. ' + subst_rule_docs
        }),
        'destination': 'global',
        'default': (
            # take any github project URL apart into <org>###<identifier>
            r',https?://github.com/([^/]+)/(.*)$,\1###\2',
            # replace any (back)slashes with a single dash
            r',[/\\]+,-',
            # replace any whitespace (include urlquoted variant)
            # with a single underscore
            r',\s+|(%2520)+|(%20)+,_',
            # rebuild functional project URL
            r',([^#]+)###(.*),https://github.com/\1/\2',
        )
    },
    # TODO this one should migrate to the datalad-osf extension. however, right
    # now extensions cannot provide default configuration
    # https://github.com/datalad/datalad/issues/5769
    'datalad.clone.url-substitute.osf': {
        'ui': ('question', {
               'title': 'Open Science Framework URL substitution rule',
               'text': 'Mangling for OSF-related URLs. ' + subst_rule_docs
        }),
        'destination': 'global',
        'default': (
            # accept browser-provided URL and convert to those accepted by
            # the datalad-osf extension
            r',^https://osf.io/([^/]+)[/]*$,osf://\1',
        )
    },
    # this is actually used in downloaders, but kept cfg name original
    'datalad.crawl.cache': {
        'ui': ('yesno', {
               'title': 'Crawler download caching',
               'text': 'Should the crawler cache downloaded files?'}),
        'destination': 'local',
        'type': bool,
    },
    'datalad.externals.nda.dbserver': {
        'ui': ('question', {
               'title': 'NDA database server',
               'text': 'Hostname of the database server'}),
        'destination': 'global',
        # Development one is https://development.nimhda.org
        'default': 'https://nda.nih.gov/DataManager/dataManager',
    },
    'datalad.locations.cache': {
        'ui': ('question', {
               'title': 'Cache directory',
               'text': 'Where should datalad cache files?'}),
        'destination': 'global',
        'default_fn': lambda: dirs.user_cache_dir,
    },
    'datalad.locations.default-dataset': {
        'ui': ('question', {
               'title': 'Default dataset path',
               'text': 'Where should datalad should look for (or install) a '
                       'default dataset?'}),
        'destination': 'global',
        'default_fn': lambda: opj(expanduser('~'), 'datalad'),
    },
    'datalad.locations.locks': {
        'ui': ('question', {
               'title': 'Lockfile directory',
               'text': 'Where should datalad store lock files?'}),
        'destination': 'global',
        'default_fn': lambda: opj(dirs.user_cache_dir, 'locks')
    },
    'datalad.locations.sockets': {
        'ui': ('question', {
               'title': 'Socket directory',
               'text': 'Where should datalad store socket files?'}),
        'destination': 'global',
        'default_fn': lambda: opj(dirs.user_cache_dir, 'sockets'),
    },
    'datalad.locations.system-procedures': {
        'ui': ('question', {
               'title': 'System procedure directory',
               'text': 'Where should datalad search for system procedures?'}),
        'destination': 'global',
        'default_fn': lambda: opj(dirs.site_config_dir, 'procedures'),
    },
    'datalad.locations.user-procedures': {
        'ui': ('question', {
               'title': 'User procedure directory',
               'text': 'Where should datalad search for user procedures?'}),
        'destination': 'global',
        'default_fn': lambda: opj(dirs.user_config_dir, 'procedures'),
    },
    'datalad.locations.extra-procedures': {
        'ui': ('question', {
            'title': 'Extra procedure directory',
            'text': 'Where should datalad search for some additional procedures?'}),
        'destination': 'global',
    },
    'datalad.locations.dataset-procedures': {
        'ui': ('question', {
               'title': 'Dataset procedure directory',
               'text': 'Where should datalad search for dataset procedures (relative to a dataset root)?'}),
        'destination': 'dataset',
        'default': opj('.datalad', 'procedures'),
    },
    'datalad.exc.str.tblimit': {
        'ui': ('question', {
               'title': 'This flag is used by datalad to cap the number of traceback steps included in exception logging and result reporting to DATALAD_EXC_STR_TBLIMIT of pre-processed entries from traceback.'}),
    },
    'datalad.fake-dates': {
        'ui': ('yesno', {
               'title': 'Fake (anonymize) dates',
               'text': 'Should the dates in the logs be faked?'}),
        'destination': 'local',
        'type': EnsureBool(),
        'default': False,
    },
    'datalad.fake-dates-start': {
        'ui': ('question', {
            'title': 'Initial fake date',
            'text': 'When faking dates and there are no commits in any local branches, generate the date by adding one second to this value (Unix epoch time). The value must be positive.'}),
        'type': EnsureInt(),
        'default': 1112911993,
    },
    'datalad.github.token-note': {
        'ui': ('question', {
            'title': 'Github token note',
            'text': 'Description for a Personal access token to generate.'}),
        'default': 'DataLad',
    },
    'datalad.tests.nonetwork': {
        'ui': ('yesno', {
               'title': 'Skips network tests completely if this flag is set Examples include test for s3, git_repositories, openfmri etc'}),
        'type': EnsureBool(),
    },
    'datalad.tests.nonlo': {
        'ui': ('question', {
               'title': 'Specifies network interfaces to bring down/up for testing. Currently used by travis.'}),
    },
    'datalad.tests.noteardown': {
        'ui': ('yesno', {
               'title': 'Does not execute teardown_package which cleans up temp files and directories created by tests if this flag is set'}),
        'type': EnsureBool(),
    },
    'datalad.tests.dataladremote': {
        'ui': ('yesno', {
               'title': 'Binary flag to specify whether each annex repository should get datalad special remote in every test repository'}),
        'type': EnsureBool(),
    },
    'datalad.tests.runcmdline': {
        'ui': ('yesno', {
               'title': 'Binary flag to specify if shell testing using shunit2 to be carried out'}),
        'type': EnsureBool(),
    },
    'datalad.tests.ssh': {
        'ui': ('yesno', {
               'title': 'Skips SSH tests if this flag is **not** set'}),
        'type': EnsureBool(),
    },
    'datalad.tests.knownfailures.skip': {
        'ui': ('yesno', {
               'title': 'Skips tests that are known to currently fail'}),
        'type': EnsureBool(),
        'default': True,
    },
    'datalad.tests.knownfailures.probe': {
        'ui': ('yesno', {
               'title': 'Probes tests that are known to fail on whether or not they are actually still failing'}),
        'type': EnsureBool(),
        'default': False,
    },
    'datalad.tests.setup.testrepos': {
        'ui': ('question', {
            'title': 'Pre-creates repositories for @with_testrepos within setup_package'}),
        'type': EnsureBool(),
        'default': False,
    },
    'datalad.tests.temp.dir': {
        'ui': ('question', {
               'title': 'Create a temporary directory at location specified by this flag. It is used by tests to create a temporary git directory while testing git annex archives etc'}),
        'type': EnsureStr(),
        'default_fn': lambda: environ.get('TMPDIR'),
    },
    'datalad.tests.temp.keep': {
        'ui': ('yesno', {
               'title': 'Function rmtemp will not remove temporary file/directory created for testing if this flag is set'}),
        'type': EnsureBool(),
    },
    'datalad.tests.temp.fs': {
        'ui': ('question', {
               'title': 'Specify the temporary file system to use as loop device for testing DATALAD_TESTS_TEMP_DIR creation'}),
    },
    'datalad.tests.temp.fssize': {
        'ui': ('question', {
               'title': 'Specify the size of temporary file system to use as loop device for testing DATALAD_TESTS_TEMP_DIR creation'}),
    },
    'datalad.tests.ui.backend': {
        'ui': ('question', {
            'title': 'Tests UI backend',
            # XXX we could add choices...
            'text': 'Which UI backend to use'}),
        'default': 'tests-noninteractive',
    },
    'datalad.tests.usecassette': {
        'ui': ('question', {
               'title': 'Specifies the location of the file to record network transactions by the VCR module. Currently used by when testing custom special remotes'}),
    },
    'datalad.tests.cache': {
        'ui': ('question', {
            'title': 'Cache directory for tests',
            'text': 'Where should datalad cache test files?'}),
        'destination': 'global',
        'default_fn': lambda: opj(dirs.user_cache_dir, 'tests')
    },
    'datalad.log.level': {
        'ui': ('question', {
            'title': 'Used for control the verbosity of logs printed to '
                     'stdout while running datalad commands/debugging'}),
    },
    'datalad.log.result-level': {
        'ui': ('question', {
               'title': 'Log level for command result messages',
               'text': "Overrides the default behavior of logging 'impossible' "
                       "results as a warning, 'error' results as errors, and "
                       "everything else as 'debug' with a single alternative "
                       "log level"}),
        'type': EnsureChoice('debug', 'info', 'warning', 'error'),
        # None keeps the default behavior
        'default': 'None',
    },
    'datalad.log.name': {
        'ui': ('question', {
            'title': 'Include name of the log target in the log line'}),
    },
    'datalad.log.names': {
        'ui': ('question', {
            'title': 'Which names (,-separated) to print log lines for'}),
    },
    'datalad.log.namesre': {
        'ui': ('question', {
            'title': 'Regular expression for which names to print log lines for'}),
    },
    'datalad.log.outputs': {
        'ui': ('question', {
               'title': 'Whether to log stdout and stderr for executed commands',
               'text': 'When enabled, setting the log level to 5 '
                       'should catch all execution output, '
                       'though some output may be logged at higher levels'}),
        'default': False,
        'type': EnsureBool(),
    },
    'datalad.log.timestamp': {
        'ui': ('yesno', {
               'title': 'Used to add timestamp to datalad logs'}),
        'default': False,
        'type': EnsureBool(),
    },
    'datalad.log.traceback': {
        'ui': ('question', {
               'title': 'Runs TraceBack function with collide set to True, if this flag is set to "collide". This replaces any common prefix between current traceback log and previous invocation with "..."'}),
    },
    'datalad.log.exc': {
        'ui': ('yesno', {
               'title': 'Include exceptions and their traceback in log messages. If set, \'datalad.exc.str.tblimit\' applies.'}),
        'default': False,
        'type': EnsureBool(),
    },
    'datalad.ssh.identityfile': {
        'ui': ('question', {
               'title': "If set, pass this file as ssh's -i option."}),
        'destination': 'global',
        'default': None,
    },
    'datalad.ssh.multiplex-connections': {
        'ui': ('question', {
               'title': "Whether to use a single shared connection for multiple SSH processes aiming at the same target."}),
        'destination': 'global',
        'default': not on_windows,
        'type': EnsureBool(),
    },
    'datalad.annex.retry': {
        'ui': ('question',
               {'title': 'Value for annex.retry to use for git-annex calls',
                'text': 'On transfer failure, annex.retry (sans "datalad.") '
                        'controls the number of times that git-annex retries. '
                        'DataLad will call git-annex with annex.retry set '
                        'to the value here unless the annex.retry '
                        'is explicitly configured'}),
        'type': EnsureInt(),
        'default': 3,
    },
    'datalad.repo.backend': {
        'ui': ('question', {
               'title': 'git-annex backend',
               'text': 'Backend to use when creating git-annex repositories'}),
        'default': 'MD5E',
    },
    'datalad.repo.direct': {
        'ui': ('yesno', {
               'title': 'Direct Mode for git-annex repositories',
               'text': 'Set this flag to create annex repositories in direct mode by default'}),
        'type': EnsureBool(),
        'default': False,
    },
    'datalad.repo.version': {
        'ui': ('question', {
               'title': 'git-annex repository version',
               'text': 'Specifies the repository version for git-annex to be used by default'}),
        'type': EnsureInt(),
        'default': 8,
    },
    'datalad.metadata.maxfieldsize': {
        'ui': ('question', {
               'title': 'Maximum metadata field size',
               'text': 'Metadata fields exceeding this size (in bytes/chars) are excluded from metadata extractio'}),
        'default': 100000,
        'type': EnsureInt(),
    },
    'datalad.metadata.nativetype': {
        'ui': ('question', {
               'title': 'Native dataset metadata scheme',
               'text': 'Set this label to engage a particular metadata extraction parser'}),
    },
    'datalad.metadata.store-aggregate-content': {
        'ui': ('question', {
               'title': 'Aggregated content metadata storage',
               'text': 'If this flag is enabled, content metadata is aggregated into superdataset to allow for discovery of individual files. If disable unique content metadata values are still aggregated to enable dataset discovery'}),
        'type': EnsureBool(),
        'default': True,
    },
    'datalad.search.default-mode': {
        'ui': ('question', {
               'title': 'Default search mode',
               'text': 'Label of the mode to be used by default'}),
        'type': EnsureChoice('egrep', 'textblob', 'autofield'),  # graph,...
        'default': 'egrep',
    },
    'datalad.search.index-default-documenttype': {
        'ui': ('question', {
               'title': 'Type of search index documents',
               'text': 'Labels of document types to include in a default search index'}),
        'type': EnsureChoice('all', 'datasets', 'files'),
        'default': 'datasets',
    },
    'datalad.metadata.create-aggregate-annex-limit': {
        'ui': ('question', {
               'title': 'Limit configuration annexing aggregated metadata in new dataset',
               'text': 'Git-annex large files expression (see https://git-annex.branchable.com/tips/largefiles; given expression will be wrapped in parentheses)'}),
        'default': 'anything',
    },
    'datalad.runtime.max-annex-jobs': {
        'ui': ('question', {
               'title': 'Maximum number of git-annex jobs to request when "jobs" option set to "auto" (default)',
               'text': 'Set this value to enable parallel annex jobs that may speed up certain operations (e.g. get file content). The effective number of jobs will not exceed the number of available CPU cores (or 3 if there is less than 3 cores).'}),
        'type': EnsureInt(),
        'default': 1,
    },
    'datalad.runtime.max-jobs': {
        'ui': ('question', {
            'title': 'Maximum number of jobs DataLad can run in "parallel"',
            'text': 'Set this value to enable parallel multi-threaded DataLad jobs that may speed up certain '
                    'operations, in particular operation across multiple datasets (e.g., install multiple '
                    'subdatasets, etc).'}),
        'type': EnsureInt(),
        'default': 1,
    },
    'datalad.runtime.raiseonerror': {
        'ui': ('question', {
               'title': 'Error behavior',
               'text': 'Set this flag to cause DataLad to raise an exception on errors that would have otherwise just get logged'}),
        'type': EnsureBool(),
        'default': False,
    },
    'datalad.runtime.report-status': {
        'ui': ('question', {
               'title': 'Command line result reporting behavior',
               'text': "If set (to other than 'all'), constrains command result report to records matching the given status. 'success' is a synonym for 'ok' OR 'notneeded', 'failure' stands for 'impossible' OR 'error'"}),
        'type': EnsureChoice('all', 'success', 'failure', 'ok', 'notneeded', 'impossible', 'error'),
        'default': None,
    },
    'datalad.runtime.stalled-external': {
        'ui': ('question', {
            'title': 'Behavior for handing external processes',
            'text': 'What to do with external processes if they do not finish in some minimal reasonable time. '
                    'If "abandon", datalad would proceed without waiting for external process to exit. '
                    'ATM applies only to batched git-annex processes. Should be changed with caution.'}),
        'type': EnsureChoice('wait', 'abandon'),
        'default': 'wait',
    },
    'datalad.search.indexercachesize': {
        'ui': ('question', {
               'title': 'Maximum cache size for search index (per process)',
               'text': 'Actual memory consumption can be twice as high as this value in MB (one process per CPU is used)'}),
        'default': 256,
        'type': EnsureInt(),
    },
    'datalad.ui.progressbar': {
        'ui': ('question', {
            'title': 'UI progress bars',
            'text': 'Default backend for progress reporting'}),
        'default': None,
        'type': EnsureChoice('tqdm', 'tqdm-ipython', 'log', 'none'),
    },
    'datalad.ui.color': {
        'ui': ('question', {
            'title': 'Colored terminal output',
            'text': 'Enable or disable ANSI color codes in outputs; "on" overrides NO_COLOR environment variable'}),
        'default': 'auto',
        'type': EnsureChoice('on', 'off', 'auto'),
    },
    'datalad.ui.suppress-similar-results': {
        'ui': ('question', {
            'title': 'Suppress rendering of similar repetitive results',
            'text': "If enabled, after a certain number of subsequent "
                    "results that are identical regarding key properties, "
                    "such as 'status', 'action', and 'type', additional "
                    "similar results are not rendered by the common result "
                    "renderer anymore. Instead, a count "
                    "of suppressed results is displayed. If disabled, or "
                    "when not running in an interactive terminal, all results "
                    "are rendered."}),
        'default': True,
        'type': EnsureBool(),
    },
    'datalad.ui.suppress-similar-results-threshold': {
        'ui': ('question', {
            'title': 'Threshold for suppressing similar repetitive results',
            'text': "Minimum number of similar results to occur before "
                    "suppression is considered. "
                    "See 'datalad.ui.suppress-similar-results' for more "
                    "information."}),
        'default': 10,
        'type': EnsureInt(),
    },
    'datalad.save.no-message': {
        'ui': ('question', {
            'title': 'Commit message handling',
            'text': 'When no commit message was provided: '
                    'attempt to obtain one interactively (interactive); '
                    'or use a generic commit message (generic). '
                    'NOTE: The interactive option is experimental. The '
                    'behavior may change in backwards-incompatible ways.'}),
        'default': 'generic',
        'type': EnsureChoice('interactive', 'generic'),
    },
    'datalad.install.inherit-local-origin': {
        'ui': ('question', {
            'title': 'Inherit local origin of dataset source',
            'text': "If enabled, a local 'origin' remote of a local dataset "
                    "clone source is configured as an 'origin-2' remote "
                    "to make its annex automatically available. The process "
                    "is repeated recursively for any further qualifying "
                    "'origin' dataset thereof."
                    "Note that if clone.defaultRemoteName is configured "
                    "to use a name other than 'origin', that name will be "
                    "used instead."}),
        'default': True,
        'type': EnsureBool(),
    },
}


def compute_cfg_defaults():
    """Compute dynamic defaults for configuration options.

    These are options that depend on things like $HOME that change under our
    testing setup.
    """
    for key, value in definitions.items():
        def_fn = value.get("default_fn")
        if def_fn:
            value['default'] = def_fn()


compute_cfg_defaults()


### datalad/downloaders/credentials.py
# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-
# ex: set sts=4 ts=4 sw=4 noet:
# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
#
#   See COPYING file distributed along with the datalad package for the
#   copyright and license terms.
#
# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
"""Interface information about credentials

Provides minimalistic interface to deal (query, request, store) with most common
types of credentials.  To be used by Authenticators
"""

__dev_doc__ = """
Possibly useful in the future 3rd part developments

https://github.com/omab/python-social-auth
    social authentication/registration mechanism with support for several
    frameworks and auth providers
"""

import time

from collections import OrderedDict

from ..dochelpers import exc_str
from ..support.exceptions import AccessDeniedError
from ..support.keyring_ import keyring as keyring_
from ..ui import ui
from ..utils import auto_repr
from ..support.network import iso8601_to_epoch

from datalad import cfg as dlcfg

from logging import getLogger
lgr = getLogger('datalad.downloaders.credentials')


@auto_repr
class Credential(object):
    """Base class for different types of credentials
    """

    # Should be defined in a subclass as an OrderedDict of fields
    # name -> dict(attributes)
    _FIELDS = None
    _KNOWN_ATTRS = {
        'hidden',    # UI should not display the value
        'repeat',    # UI should repeat entry or not. Set to False to override default logic
        'optional',  # Not mandatory thus not requested if not set
    }

    def __init__(self, name, url=None, keyring=None):
        """
        Parameters
        ----------
        name : str
            Name of the credential, as it would be identified by in the centralized
            storage of credentials
        url : str, optional
            URL string to point users to where to seek obtaining the credentials
        keyring : a keyring
            An object providing (g|s)et_password.  If None, keyring module is used
            as is
        """
        self.name = name
        self.url = url
        self._keyring = keyring or keyring_
        self._prepare()

    def _prepare(self):
        """Additional house-keeping possibly to be performed in subclasses

        Created to avoid all the passing args/kwargs of __init__ all the time
        """
        # Basic checks
        for f, fattrs in self._FIELDS.items():
            unknown_attrs = set(fattrs).difference(self._KNOWN_ATTRS)
            if unknown_attrs:
                raise ValueError("Unknown attributes %s. Known are: %s"
                                 % (unknown_attrs, self._KNOWN_ATTRS))

    def _is_field_optional(self, f):
        return self._FIELDS[f].get('optional', False)

    @property
    def is_known(self):
        """Return True if values for all fields of the credential are known"""
        try:
            return all(
                self._is_field_optional(f) or self._get_field_value(f) is not None
                for f in self._FIELDS)
        except Exception as exc:
            lgr.warning("Failed to query keyring: %s" % exc_str(exc))
            return False

    def _get_field_value(self, field):
        return dlcfg.get('datalad.credential.{name}.{field}'.format(
            name=self.name,
            field=field.replace('_', '-')
        )) or self._keyring.get(self.name, field)

    def _ask_field_value(self, f, instructions=None):
        msg = instructions if instructions else \
            ("You need to authenticate with %r credentials." % self.name +
                  (" %s provides information on how to gain access"
                   % self.url if self.url else ''))

        # provide custom options only if set for the field
        f_props = self._FIELDS[f]
        kwargs = {}
        for p in ('hidden', 'repeat'):
            if p in f_props:
                kwargs[p] = f_props[p]
        return ui.question(
            f,
            title=msg,
            **kwargs
        )

    def _ask_and_set(self, f, instructions=None):
        v = self._ask_field_value(f, instructions=instructions)
        self.set(**{f: v})
        return v

    def enter_new(self, instructions=None, **kwargs):
        """Enter new values for the credential fields

        Parameters
        ----------
        instructions : str, optional
          If given, the auto-generated instructions based on a login-URL are
          replaced by the given string
        **kwargs
          Any given key value pairs with non-None values are used to set the
          field `key` to the given value, without asking for user input
        """
        if kwargs:
            unknown_fields = set(kwargs).difference(self._FIELDS)
            known_fields = set(self._FIELDS).difference(kwargs)
            if unknown_fields:
                raise ValueError(
                    "Unknown to %s field(s): %s.  Known but not specified: %s"
                    % (self,
                       ', '.join(sorted(unknown_fields)),
                       ', '.join(sorted(known_fields))
                       ))
        # Use ui., request credential fields corresponding to the type
        for f in self._FIELDS:
            if kwargs.get(f, None):
                # use given value, don't ask
                self.set(**{f: kwargs[f]})
            elif not self._is_field_optional(f):
                self._ask_and_set(f, instructions=instructions)

    def __call__(self):
        """Obtain credentials from a keyring and if any is not known -- ask"""
        fields = {}
        for f in self._FIELDS:
            v = self._get_field_value(f)
            if not self._is_field_optional(f):
                while v is None:  # was not known
                    v = self._ask_and_set(f)
                fields[f] = v
            elif v is not None:
                fields[f] = v
        return fields

    def set(self, **kwargs):
        """Set field(s) of the credential"""
        for f, v in kwargs.items():
            if f not in self._FIELDS:
                raise ValueError("Unknown field %s. Known are: %s"
                                 % (f, self._FIELDS.keys()))
            self._keyring.set(self.name, f, v)

    def get(self, f, default=None):
        """Get a field of the credential"""
        if f not in self._FIELDS:
            raise ValueError("Unknown field %s. Known are: %s"
                             % (f, self._FIELDS.keys()))
        try:
            return self._get_field_value(f)
        except:  # MIH: what could even happen? _keyring not a dict?
            return default

    def delete(self):
        """Deletes credential values from the keyring"""
        for f in self._FIELDS:
            self._keyring.delete(self.name, f)


class UserPassword(Credential):
    """Simple type of a credential which consists of user/password pair"""

    _FIELDS = OrderedDict([('user', {}),
                           ('password', {'hidden': True})])

    is_expired = False  # no expiration provisioned


class Token(Credential):
    """Simple type of a credential which provides a single token"""

    _FIELDS = OrderedDict([('token', {'hidden': True, 'repeat': False})])

    is_expired = False  # no expiration provisioned


class AWS_S3(Credential):
    """Credential for AWS S3 service"""

    _FIELDS = OrderedDict([('key_id', {'repeat': False}),
                           ('secret_id', {'hidden': True, 'repeat': False}),
                           ('session', {'optional': True}),
                           ('expiration', {'optional': True}),
                           ])

    @property
    def is_expired(self):
        exp = self.get('expiration', None)
        if not exp:
            return False
        exp_epoch = iso8601_to_epoch(exp)
        # -2 to artificially shorten duration of the allotment to avoid
        # possible race conditions between us checking either it has
        # already expired before submitting a request.
        expire_in = (exp_epoch - time.time() - 2) / 3600.

        lgr.debug(
            ("Credential %s has expired %.2fh ago"
                if expire_in <= 0 else "Credential %s will expire in %.2fh")
            % (self.name, expire_in))
        return expire_in <= 0


@auto_repr
class CompositeCredential(Credential):
    """Credential which represent a sequence of Credentials where front one is exposed to user
    """

    # To be defined in sub-classes
    _CREDENTIAL_CLASSES = None
    _CREDENTIAL_ADAPTERS = None

    def _prepare(self):
        assert len(self._CREDENTIAL_CLASSES) > 1, "makes sense only if there is > 1 credential"
        assert len(self._CREDENTIAL_CLASSES) == len(self._CREDENTIAL_ADAPTERS) + 1, \
            "there should be 1 less of adapter than _CREDENTIAL_CLASSES"

        for C in self._CREDENTIAL_CLASSES:
            assert issubclass(C, Credential), "%s must be a subclass of Credential" % C

        # First credential should bear the name and url
        credentials = [self._CREDENTIAL_CLASSES[0](self.name, url=self.url, keyring=self._keyring)]
        # and we just reuse its _FIELDS for _ask_field_value etc
        self._FIELDS = credentials[0]._FIELDS
        # the rest with index suffix, but storing themselves in the same keyring
        for iC, C in enumerate(self._CREDENTIAL_CLASSES[1:]):
            credentials.append(
                C(name="%s:%d" % (self.name, iC + 1), url=None, keyring=self._keyring)
            )
        self._credentials = credentials

        super(CompositeCredential, self)._prepare()

    # Here it becomes tricky, since theoretically it is the "tail"
    # ones which might expire etc, so we wouldn't exactly know what
    # new credentials outside process wanted -- it would be silly to ask
    # right away the "entry" credentials if it is just the expiration of the
    # tail credentials
    def enter_new(self):
        # should invalidate/remove all tail credentials to avoid failing attempts to login
        self._credentials[0].enter_new()
        self.refresh()

    def refresh(self):
        """Re-establish "dependent" credentials

        E.g. if code outside was reported that it expired somehow before known expiration datetime
        """
        for c in self._credentials[1:]:
            c.delete()
        # trigger re-establishing the chain
        _ = self()
        if self.is_expired:
            raise RuntimeError("Credential %s expired right upon refresh: should have not happened")

    @property
    def is_expired(self):
        return any(c.is_expired for c in self._credentials)

    def __call__(self):
        """Obtain credentials from a keyring and if any is not known -- ask"""
        # Start from the tail until we have credentials set
        idx = len(self._credentials) - 1
        for c in self._credentials[::-1]:
            if c.is_known and not c.is_expired:
                break
            idx -= 1

        if idx < 0:
            # none was known, all the same -- start with the first one
            idx = 0

        # TODO: consider moving logic of traversal into authenticator since it is
        # the one spitting out authentication error etc
        # Theoretically we could just reuse 'fields' from adapter in the next step
        # but let's do full circle, so that if any "normalization" is done by
        # Credentials we take that into account
        for c, adapter, next_c in zip(
                self._credentials[idx:],
                self._CREDENTIAL_ADAPTERS[idx:],
                self._credentials[idx + 1:]):
            fields = c()
            next_fields = adapter(self, **fields)
            next_c.set(**next_fields)

        return self._credentials[-1]()


def _nda_adapter(composite, user=None, password=None):
    from datalad.support.third.nda_aws_token_generator import NDATokenGenerator
    from .. import cfg
    nda_auth_url = cfg.obtain('datalad.externals.nda.dbserver')
    gen = NDATokenGenerator(nda_auth_url)
    lgr.debug("Generating token for NDA user %s using %s talking to %s",
              user, gen, nda_auth_url)
    try:
        token = gen.generate_token(user, password)
    except Exception as exc:  # it is really just an "Exception"
        exc_str = str(exc).lower()
        # ATM it is "Invalid username and/or password"
        # but who knows what future would bring
        if "invalid" in exc_str and ("user" in exc_str or "password" in exc_str):
            raise AccessDeniedError(exc_str)
        raise
    # There are also session and expiration we ignore... TODO anything about it?!!!
    # we could create a derived AWS_S3 which would also store session and expiration
    # and then may be Composite could use those????
    return dict(key_id=token.access_key, secret_id=token.secret_key,
                session=token.session, expiration=token.expiration)


class NDA_S3(CompositeCredential):
    """Credential to access NDA AWS

    So for NDA we need a credential which is a composite credential.
    User provides UserPassword and then some adapter generates AWS_S3
    out of it
    """
    _CREDENTIAL_CLASSES = (UserPassword, AWS_S3)
    _CREDENTIAL_ADAPTERS = (_nda_adapter,)


def _loris_adapter(composite, user=None, password=None, **kwargs):
    from datalad.support.third.loris_token_generator import LORISTokenGenerator

    gen = LORISTokenGenerator(url=composite.url)
    token = gen.generate_token(user, password)

    return dict(token=token)


class LORIS_Token(CompositeCredential):
    _CREDENTIAL_CLASSES = (UserPassword, Token)
    _CREDENTIAL_ADAPTERS = (_loris_adapter,)

    def __init__(self, name, url=None, keyring=None):
        super(CompositeCredential, self).__init__(name, url, keyring)


CREDENTIAL_TYPES = {
    'user_password': UserPassword,
    'aws-s3': AWS_S3,
    'nda-s3': NDA_S3,
    'token': Token,
    'loris-token': LORIS_Token
}

```
--- END FILE ---

Please first localize the bug based on the issue statement, and then generate *SEARCH/REPLACE* edits to fix the issue.

Every *SEARCH/REPLACE* edit must use this format:
1. The file path
2. The start of search block: <<<<<<< SEARCH
3. A contiguous chunk of lines to search for in the existing source code
4. The dividing line: =======
5. The lines to replace into the source code
6. The end of the replace block: >>>>>>> REPLACE

Here is an example:

```python
### mathweb/flask/app.py
<<<<<<< SEARCH
from flask import Flask
=======
import math
from flask import Flask
>>>>>>> REPLACE
```
</solution>

Please note that the *SEARCH/REPLACE* edit REQUIRES PROPER INDENTATION. If you would like to add the line '        print(x)', you must fully write that out, with all those spaces before the code!
Wrap each *SEARCH/REPLACE* edit in a code block as shown in the example above. If you have multiple *SEARCH/REPLACE* edits, use a separate code block for each one.
[ISSUE]

**Title:** Forced Credential Re-entry Not Triggered with `datalad.credentials.force-ask` Configuration

**Description:**

When attempting to update stored credentials by enabling the `datalad.credentials.force-ask` configuration setting, DataLad does not prompt for credential re-entry. This prevents users from updating their credentials seamlessly, as the existing credentials remain unchanged.

**Example Code:**

```python
from datalad.downloaders.credentials import UserPassword
from datalad import cfg as dlcfg

# Initialize keyring and set initial credentials
keyring = MemoryKeyring()
cred = UserPassword("name", keyring=keyring)
keyring.set('name', 'user', 'user1')
keyring.set('name', 'password', 'password1')

# Retrieve existing credentials
credentials = cred()
print(credentials)  # Outputs: {'user': 'user1', 'password': 'password1'}

# Attempt to force credential re-entry
dlcfg.set('datalad.credentials.force-ask', True)
credentials = cred()
print(credentials)  # Expected to prompt for new credentials and update
```

**Expected Behavior:**

When `datalad.credentials.force-ask` is set to `True`, calling `cred()` should prompt the user to re-enter their credentials. The new credentials provided by the user should then replace the existing ones in the keyring.

**Actual Behavior:**

Despite setting `force-ask` to `True`, calling `cred()` does not prompt for re-entry. The original credentials `{'user': 'user1', 'password': 'password1'}` are returned without any update, making it impossible to update the stored credentials through this configuration.

[/ISSUE]
{
    "test_cred1_enter_new": "PASSED",
    "test_cred1_call": "PASSED",
    "test_composite_credential1": "PASSED",
    "test_credentials_from_env": "PASSED"
}
diff --git a/datalad/downloaders/credentials.py b/datalad/downloaders/credentials.py
index 063038936..21aaea39a 100644
--- a/datalad/downloaders/credentials.py
+++ b/datalad/downloaders/credentials.py
@@ -32,6 +32,7 @@ from ..utils import auto_repr
 from ..support.network import iso8601_to_epoch
 
 from datalad import cfg as dlcfg
+from datalad.config import anything2bool
 
 from logging import getLogger
 lgr = getLogger('datalad.downloaders.credentials')
@@ -157,8 +158,14 @@ class Credential(object):
     def __call__(self):
         """Obtain credentials from a keyring and if any is not known -- ask"""
         fields = {}
+        # check if we shall ask for credentials, even if some are on record
+        # already (but maybe they were found to need updating)
+        force_reentry = dlcfg.obtain(
+            'datalad.credentials.force-ask',
+            valtype=anything2bool)
         for f in self._FIELDS:
-            v = self._get_field_value(f)
+            # don't query for value if we need to get a new one
+            v = None if force_reentry else self._get_field_value(f)
             if not self._is_field_optional(f):
                 while v is None:  # was not known
                     v = self._ask_and_set(f)
diff --git a/datalad/interface/common_cfg.py b/datalad/interface/common_cfg.py
index 7485cf577..58d9d49a5 100644
--- a/datalad/interface/common_cfg.py
+++ b/datalad/interface/common_cfg.py
@@ -80,6 +80,15 @@ definitions = {
         'destination': 'local',
         'type': bool,
     },
+    # this is actually used in downloaders, but kept cfg name original
+    'datalad.credentials.force-ask': {
+        'ui': ('yesno', {
+               'title': 'Force (re-)entry of credentials',
+               'text': 'Should DataLad prompt for credential (re-)entry? This '
+                       'can be used to update previously stored credentials.'}),
+        'type': bool,
+        'default': False,
+    },
     'datalad.externals.nda.dbserver': {
         'ui': ('question', {
                'title': 'NDA database server',
 

{"datalad/downloaders/credentials.py": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Interface information about credentials\n\nProvides minimalistic interface to deal (query, request, store) with most common\ntypes of credentials.  To be used by Authenticators\n\"\"\"\n\n__dev_doc__ = \"\"\"\nPossibly useful in the future 3rd part developments\n\nhttps://github.com/omab/python-social-auth\n    social authentication/registration mechanism with support for several\n    frameworks and auth providers\n\"\"\"\n\nimport time\n\nfrom collections import OrderedDict\n\nfrom ..dochelpers import exc_str\nfrom ..support.exceptions import AccessDeniedError\nfrom ..support.keyring_ import keyring as keyring_\nfrom ..ui import ui\nfrom ..utils import auto_repr\nfrom ..support.network import iso8601_to_epoch\n\nfrom datalad import cfg as dlcfg\n\nfrom logging import getLogger\nlgr = getLogger('datalad.downloaders.credentials')\n\n\n@auto_repr\nclass Credential(object):\n    \"\"\"Base class for different types of credentials\n    \"\"\"\n\n    # Should be defined in a subclass as an OrderedDict of fields\n    # name -> dict(attributes)\n    _FIELDS = None\n    _KNOWN_ATTRS = {\n        'hidden',    # UI should not display the value\n        'repeat',    # UI should repeat entry or not. Set to False to override default logic\n        'optional',  # Not mandatory thus not requested if not set\n    }\n\n    def __init__(self, name, url=None, keyring=None):\n        \"\"\"\n        Parameters\n        ----------\n        name : str\n            Name of the credential, as it would be identified by in the centralized\n            storage of credentials\n        url : str, optional\n            URL string to point users to where to seek obtaining the credentials\n        keyring : a keyring\n            An object providing (g|s)et_password.  If None, keyring module is used\n            as is\n        \"\"\"\n        self.name = name\n        self.url = url\n        self._keyring = keyring or keyring_\n        self._prepare()\n\n    def _prepare(self):\n        \"\"\"Additional house-keeping possibly to be performed in subclasses\n\n        Created to avoid all the passing args/kwargs of __init__ all the time\n        \"\"\"\n        # Basic checks\n        for f, fattrs in self._FIELDS.items():\n            unknown_attrs = set(fattrs).difference(self._KNOWN_ATTRS)\n            if unknown_attrs:\n                raise ValueError(\"Unknown attributes %s. Known are: %s\"\n                                 % (unknown_attrs, self._KNOWN_ATTRS))\n\n    def _is_field_optional(self, f):\n        return self._FIELDS[f].get('optional', False)\n\n    @property\n    def is_known(self):\n        \"\"\"Return True if values for all fields of the credential are known\"\"\"\n        try:\n            return all(\n                self._is_field_optional(f) or self._get_field_value(f) is not None\n                for f in self._FIELDS)\n        except Exception as exc:\n            lgr.warning(\"Failed to query keyring: %s\" % exc_str(exc))\n            return False\n\n    def _get_field_value(self, field):\n        return dlcfg.get('datalad.credential.{name}.{field}'.format(\n            name=self.name,\n            field=field.replace('_', '-')\n        )) or self._keyring.get(self.name, field)\n\n    def _ask_field_value(self, f, instructions=None):\n        msg = instructions if instructions else \\\n            (\"You need to authenticate with %r credentials.\" % self.name +\n                  (\" %s provides information on how to gain access\"\n                   % self.url if self.url else ''))\n\n        # provide custom options only if set for the field\n        f_props = self._FIELDS[f]\n        kwargs = {}\n        for p in ('hidden', 'repeat'):\n            if p in f_props:\n                kwargs[p] = f_props[p]\n        return ui.question(\n            f,\n            title=msg,\n            **kwargs\n        )\n\n    def _ask_and_set(self, f, instructions=None):\n        v = self._ask_field_value(f, instructions=instructions)\n        self.set(**{f: v})\n        return v\n\n    def enter_new(self, instructions=None, **kwargs):\n        \"\"\"Enter new values for the credential fields\n\n        Parameters\n        ----------\n        instructions : str, optional\n          If given, the auto-generated instructions based on a login-URL are\n          replaced by the given string\n        **kwargs\n          Any given key value pairs with non-None values are used to set the\n          field `key` to the given value, without asking for user input\n        \"\"\"\n        if kwargs:\n            unknown_fields = set(kwargs).difference(self._FIELDS)\n            known_fields = set(self._FIELDS).difference(kwargs)\n            if unknown_fields:\n                raise ValueError(\n                    \"Unknown to %s field(s): %s.  Known but not specified: %s\"\n                    % (self,\n                       ', '.join(sorted(unknown_fields)),\n                       ', '.join(sorted(known_fields))\n                       ))\n        # Use ui., request credential fields corresponding to the type\n        for f in self._FIELDS:\n            if kwargs.get(f, None):\n                # use given value, don't ask\n                self.set(**{f: kwargs[f]})\n            elif not self._is_field_optional(f):\n                self._ask_and_set(f, instructions=instructions)\n\n    def __call__(self):\n        \"\"\"Obtain credentials from a keyring and if any is not known -- ask\"\"\"\n        fields = {}\n        for f in self._FIELDS:\n            v = self._get_field_value(f)\n            if not self._is_field_optional(f):\n                while v is None:  # was not known\n                    v = self._ask_and_set(f)\n                fields[f] = v\n            elif v is not None:\n                fields[f] = v\n        return fields\n\n    def set(self, **kwargs):\n        \"\"\"Set field(s) of the credential\"\"\"\n        for f, v in kwargs.items():\n            if f not in self._FIELDS:\n                raise ValueError(\"Unknown field %s. Known are: %s\"\n                                 % (f, self._FIELDS.keys()))\n            self._keyring.set(self.name, f, v)\n\n    def get(self, f, default=None):\n        \"\"\"Get a field of the credential\"\"\"\n        if f not in self._FIELDS:\n            raise ValueError(\"Unknown field %s. Known are: %s\"\n                             % (f, self._FIELDS.keys()))\n        try:\n            return self._get_field_value(f)\n        except:  # MIH: what could even happen? _keyring not a dict?\n            return default\n\n    def delete(self):\n        \"\"\"Deletes credential values from the keyring\"\"\"\n        for f in self._FIELDS:\n            self._keyring.delete(self.name, f)\n\n\nclass UserPassword(Credential):\n    \"\"\"Simple type of a credential which consists of user/password pair\"\"\"\n\n    _FIELDS = OrderedDict([('user', {}),\n                           ('password', {'hidden': True})])\n\n    is_expired = False  # no expiration provisioned\n\n\nclass Token(Credential):\n    \"\"\"Simple type of a credential which provides a single token\"\"\"\n\n    _FIELDS = OrderedDict([('token', {'hidden': True, 'repeat': False})])\n\n    is_expired = False  # no expiration provisioned\n\n\nclass AWS_S3(Credential):\n    \"\"\"Credential for AWS S3 service\"\"\"\n\n    _FIELDS = OrderedDict([('key_id', {'repeat': False}),\n                           ('secret_id', {'hidden': True, 'repeat': False}),\n                           ('session', {'optional': True}),\n                           ('expiration', {'optional': True}),\n                           ])\n\n    @property\n    def is_expired(self):\n        exp = self.get('expiration', None)\n        if not exp:\n            return False\n        exp_epoch = iso8601_to_epoch(exp)\n        # -2 to artificially shorten duration of the allotment to avoid\n        # possible race conditions between us checking either it has\n        # already expired before submitting a request.\n        expire_in = (exp_epoch - time.time() - 2) / 3600.\n\n        lgr.debug(\n            (\"Credential %s has expired %.2fh ago\"\n                if expire_in <= 0 else \"Credential %s will expire in %.2fh\")\n            % (self.name, expire_in))\n        return expire_in <= 0\n\n\n@auto_repr\nclass CompositeCredential(Credential):\n    \"\"\"Credential which represent a sequence of Credentials where front one is exposed to user\n    \"\"\"\n\n    # To be defined in sub-classes\n    _CREDENTIAL_CLASSES = None\n    _CREDENTIAL_ADAPTERS = None\n\n    def _prepare(self):\n        assert len(self._CREDENTIAL_CLASSES) > 1, \"makes sense only if there is > 1 credential\"\n        assert len(self._CREDENTIAL_CLASSES) == len(self._CREDENTIAL_ADAPTERS) + 1, \\\n            \"there should be 1 less of adapter than _CREDENTIAL_CLASSES\"\n\n        for C in self._CREDENTIAL_CLASSES:\n            assert issubclass(C, Credential), \"%s must be a subclass of Credential\" % C\n\n        # First credential should bear the name and url\n        credentials = [self._CREDENTIAL_CLASSES[0](self.name, url=self.url, keyring=self._keyring)]\n        # and we just reuse its _FIELDS for _ask_field_value etc\n        self._FIELDS = credentials[0]._FIELDS\n        # the rest with index suffix, but storing themselves in the same keyring\n        for iC, C in enumerate(self._CREDENTIAL_CLASSES[1:]):\n            credentials.append(\n                C(name=\"%s:%d\" % (self.name, iC + 1), url=None, keyring=self._keyring)\n            )\n        self._credentials = credentials\n\n        super(CompositeCredential, self)._prepare()\n\n    # Here it becomes tricky, since theoretically it is the \"tail\"\n    # ones which might expire etc, so we wouldn't exactly know what\n    # new credentials outside process wanted -- it would be silly to ask\n    # right away the \"entry\" credentials if it is just the expiration of the\n    # tail credentials\n    def enter_new(self):\n        # should invalidate/remove all tail credentials to avoid failing attempts to login\n        self._credentials[0].enter_new()\n        self.refresh()\n\n    def refresh(self):\n        \"\"\"Re-establish \"dependent\" credentials\n\n        E.g. if code outside was reported that it expired somehow before known expiration datetime\n        \"\"\"\n        for c in self._credentials[1:]:\n            c.delete()\n        # trigger re-establishing the chain\n        _ = self()\n        if self.is_expired:\n            raise RuntimeError(\"Credential %s expired right upon refresh: should have not happened\")\n\n    @property\n    def is_expired(self):\n        return any(c.is_expired for c in self._credentials)\n\n    def __call__(self):\n        \"\"\"Obtain credentials from a keyring and if any is not known -- ask\"\"\"\n        # Start from the tail until we have credentials set\n        idx = len(self._credentials) - 1\n        for c in self._credentials[::-1]:\n            if c.is_known and not c.is_expired:\n                break\n            idx -= 1\n\n        if idx < 0:\n            # none was known, all the same -- start with the first one\n            idx = 0\n\n        # TODO: consider moving logic of traversal into authenticator since it is\n        # the one spitting out authentication error etc\n        # Theoretically we could just reuse 'fields' from adapter in the next step\n        # but let's do full circle, so that if any \"normalization\" is done by\n        # Credentials we take that into account\n        for c, adapter, next_c in zip(\n                self._credentials[idx:],\n                self._CREDENTIAL_ADAPTERS[idx:],\n                self._credentials[idx + 1:]):\n            fields = c()\n            next_fields = adapter(self, **fields)\n            next_c.set(**next_fields)\n\n        return self._credentials[-1]()\n\n\ndef _nda_adapter(composite, user=None, password=None):\n    from datalad.support.third.nda_aws_token_generator import NDATokenGenerator\n    from .. import cfg\n    nda_auth_url = cfg.obtain('datalad.externals.nda.dbserver')\n    gen = NDATokenGenerator(nda_auth_url)\n    lgr.debug(\"Generating token for NDA user %s using %s talking to %s\",\n              user, gen, nda_auth_url)\n    try:\n        token = gen.generate_token(user, password)\n    except Exception as exc:  # it is really just an \"Exception\"\n        exc_str = str(exc).lower()\n        # ATM it is \"Invalid username and/or password\"\n        # but who knows what future would bring\n        if \"invalid\" in exc_str and (\"user\" in exc_str or \"password\" in exc_str):\n            raise AccessDeniedError(exc_str)\n        raise\n    # There are also session and expiration we ignore... TODO anything about it?!!!\n    # we could create a derived AWS_S3 which would also store session and expiration\n    # and then may be Composite could use those????\n    return dict(key_id=token.access_key, secret_id=token.secret_key,\n                session=token.session, expiration=token.expiration)\n\n\nclass NDA_S3(CompositeCredential):\n    \"\"\"Credential to access NDA AWS\n\n    So for NDA we need a credential which is a composite credential.\n    User provides UserPassword and then some adapter generates AWS_S3\n    out of it\n    \"\"\"\n    _CREDENTIAL_CLASSES = (UserPassword, AWS_S3)\n    _CREDENTIAL_ADAPTERS = (_nda_adapter,)\n\n\ndef _loris_adapter(composite, user=None, password=None, **kwargs):\n    from datalad.support.third.loris_token_generator import LORISTokenGenerator\n\n    gen = LORISTokenGenerator(url=composite.url)\n    token = gen.generate_token(user, password)\n\n    return dict(token=token)\n\n\nclass LORIS_Token(CompositeCredential):\n    _CREDENTIAL_CLASSES = (UserPassword, Token)\n    _CREDENTIAL_ADAPTERS = (_loris_adapter,)\n\n    def __init__(self, name, url=None, keyring=None):\n        super(CompositeCredential, self).__init__(name, url, keyring)\n\n\nCREDENTIAL_TYPES = {\n    'user_password': UserPassword,\n    'aws-s3': AWS_S3,\n    'nda-s3': NDA_S3,\n    'token': Token,\n    'loris-token': LORIS_Token\n}\n", "datalad/interface/common_cfg.py": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Common configuration options\n\n\"\"\"\n\n__docformat__ = 'restructuredtext'\n\nfrom appdirs import AppDirs\nfrom os import environ\nfrom os.path import join as opj, expanduser\nfrom datalad.support.constraints import EnsureBool\nfrom datalad.support.constraints import EnsureInt\nfrom datalad.support.constraints import EnsureNone\nfrom datalad.support.constraints import EnsureChoice\nfrom datalad.support.constraints import EnsureListOf\nfrom datalad.support.constraints import EnsureStr\nfrom datalad.utils import on_windows\n\ndirs = AppDirs(\"datalad\", \"datalad.org\")\n\nsubst_rule_docs = \"\"\"\\\nA substitution specification is a string with a match and substitution\nexpression, each following Python's regular expression syntax. Both expressions\nare concatenated to a single string with an arbitrary delimiter character. The\ndelimiter is defined by prefixing the string with the delimiter. Prefix and\ndelimiter are stripped from the expressions (Example:\n\",^http://(.*)$,https://\\\\1\").  This setting can be defined multiple times.\nSubstitutions will be applied incrementally, in order of their definition. The\nfirst substitution in such a series must match, otherwise no further\nsubstitutions in a series will be considered. However, following the first\nmatch all further substitutions in a series are processed, regardless whether\nintermediate expressions match or not.\"\"\"\n\ndefinitions = {\n    'datalad.clone.url-substitute.github': {\n        'ui': ('question', {\n               'title': 'GitHub URL substitution rule',\n               'text': 'Mangling for GitHub-related URL. ' + subst_rule_docs\n        }),\n        'destination': 'global',\n        'default': (\n            # take any github project URL apart into <org>###<identifier>\n            r',https?://github.com/([^/]+)/(.*)$,\\1###\\2',\n            # replace any (back)slashes with a single dash\n            r',[/\\\\]+,-',\n            # replace any whitespace (include urlquoted variant)\n            # with a single underscore\n            r',\\s+|(%2520)+|(%20)+,_',\n            # rebuild functional project URL\n            r',([^#]+)###(.*),https://github.com/\\1/\\2',\n        )\n    },\n    # TODO this one should migrate to the datalad-osf extension. however, right\n    # now extensions cannot provide default configuration\n    # https://github.com/datalad/datalad/issues/5769\n    'datalad.clone.url-substitute.osf': {\n        'ui': ('question', {\n               'title': 'Open Science Framework URL substitution rule',\n               'text': 'Mangling for OSF-related URLs. ' + subst_rule_docs\n        }),\n        'destination': 'global',\n        'default': (\n            # accept browser-provided URL and convert to those accepted by\n            # the datalad-osf extension\n            r',^https://osf.io/([^/]+)[/]*$,osf://\\1',\n        )\n    },\n    # this is actually used in downloaders, but kept cfg name original\n    'datalad.crawl.cache': {\n        'ui': ('yesno', {\n               'title': 'Crawler download caching',\n               'text': 'Should the crawler cache downloaded files?'}),\n        'destination': 'local',\n        'type': bool,\n    },\n    'datalad.externals.nda.dbserver': {\n        'ui': ('question', {\n               'title': 'NDA database server',\n               'text': 'Hostname of the database server'}),\n        'destination': 'global',\n        # Development one is https://development.nimhda.org\n        'default': 'https://nda.nih.gov/DataManager/dataManager',\n    },\n    'datalad.locations.cache': {\n        'ui': ('question', {\n               'title': 'Cache directory',\n               'text': 'Where should datalad cache files?'}),\n        'destination': 'global',\n        'default_fn': lambda: dirs.user_cache_dir,\n    },\n    'datalad.locations.default-dataset': {\n        'ui': ('question', {\n               'title': 'Default dataset path',\n               'text': 'Where should datalad should look for (or install) a '\n                       'default dataset?'}),\n        'destination': 'global',\n        'default_fn': lambda: opj(expanduser('~'), 'datalad'),\n    },\n    'datalad.locations.locks': {\n        'ui': ('question', {\n               'title': 'Lockfile directory',\n               'text': 'Where should datalad store lock files?'}),\n        'destination': 'global',\n        'default_fn': lambda: opj(dirs.user_cache_dir, 'locks')\n    },\n    'datalad.locations.sockets': {\n        'ui': ('question', {\n               'title': 'Socket directory',\n               'text': 'Where should datalad store socket files?'}),\n        'destination': 'global',\n        'default_fn': lambda: opj(dirs.user_cache_dir, 'sockets'),\n    },\n    'datalad.locations.system-procedures': {\n        'ui': ('question', {\n               'title': 'System procedure directory',\n               'text': 'Where should datalad search for system procedures?'}),\n        'destination': 'global',\n        'default_fn': lambda: opj(dirs.site_config_dir, 'procedures'),\n    },\n    'datalad.locations.user-procedures': {\n        'ui': ('question', {\n               'title': 'User procedure directory',\n               'text': 'Where should datalad search for user procedures?'}),\n        'destination': 'global',\n        'default_fn': lambda: opj(dirs.user_config_dir, 'procedures'),\n    },\n    'datalad.locations.extra-procedures': {\n        'ui': ('question', {\n            'title': 'Extra procedure directory',\n            'text': 'Where should datalad search for some additional procedures?'}),\n        'destination': 'global',\n    },\n    'datalad.locations.dataset-procedures': {\n        'ui': ('question', {\n               'title': 'Dataset procedure directory',\n               'text': 'Where should datalad search for dataset procedures (relative to a dataset root)?'}),\n        'destination': 'dataset',\n        'default': opj('.datalad', 'procedures'),\n    },\n    'datalad.exc.str.tblimit': {\n        'ui': ('question', {\n               'title': 'This flag is used by datalad to cap the number of traceback steps included in exception logging and result reporting to DATALAD_EXC_STR_TBLIMIT of pre-processed entries from traceback.'}),\n    },\n    'datalad.fake-dates': {\n        'ui': ('yesno', {\n               'title': 'Fake (anonymize) dates',\n               'text': 'Should the dates in the logs be faked?'}),\n        'destination': 'local',\n        'type': EnsureBool(),\n        'default': False,\n    },\n    'datalad.fake-dates-start': {\n        'ui': ('question', {\n            'title': 'Initial fake date',\n            'text': 'When faking dates and there are no commits in any local branches, generate the date by adding one second to this value (Unix epoch time). The value must be positive.'}),\n        'type': EnsureInt(),\n        'default': 1112911993,\n    },\n    'datalad.github.token-note': {\n        'ui': ('question', {\n            'title': 'Github token note',\n            'text': 'Description for a Personal access token to generate.'}),\n        'default': 'DataLad',\n    },\n    'datalad.tests.nonetwork': {\n        'ui': ('yesno', {\n               'title': 'Skips network tests completely if this flag is set Examples include test for s3, git_repositories, openfmri etc'}),\n        'type': EnsureBool(),\n    },\n    'datalad.tests.nonlo': {\n        'ui': ('question', {\n               'title': 'Specifies network interfaces to bring down/up for testing. Currently used by travis.'}),\n    },\n    'datalad.tests.noteardown': {\n        'ui': ('yesno', {\n               'title': 'Does not execute teardown_package which cleans up temp files and directories created by tests if this flag is set'}),\n        'type': EnsureBool(),\n    },\n    'datalad.tests.dataladremote': {\n        'ui': ('yesno', {\n               'title': 'Binary flag to specify whether each annex repository should get datalad special remote in every test repository'}),\n        'type': EnsureBool(),\n    },\n    'datalad.tests.runcmdline': {\n        'ui': ('yesno', {\n               'title': 'Binary flag to specify if shell testing using shunit2 to be carried out'}),\n        'type': EnsureBool(),\n    },\n    'datalad.tests.ssh': {\n        'ui': ('yesno', {\n               'title': 'Skips SSH tests if this flag is **not** set'}),\n        'type': EnsureBool(),\n    },\n    'datalad.tests.knownfailures.skip': {\n        'ui': ('yesno', {\n               'title': 'Skips tests that are known to currently fail'}),\n        'type': EnsureBool(),\n        'default': True,\n    },\n    'datalad.tests.knownfailures.probe': {\n        'ui': ('yesno', {\n               'title': 'Probes tests that are known to fail on whether or not they are actually still failing'}),\n        'type': EnsureBool(),\n        'default': False,\n    },\n    'datalad.tests.setup.testrepos': {\n        'ui': ('question', {\n            'title': 'Pre-creates repositories for @with_testrepos within setup_package'}),\n        'type': EnsureBool(),\n        'default': False,\n    },\n    'datalad.tests.temp.dir': {\n        'ui': ('question', {\n               'title': 'Create a temporary directory at location specified by this flag. It is used by tests to create a temporary git directory while testing git annex archives etc'}),\n        'type': EnsureStr(),\n        'default_fn': lambda: environ.get('TMPDIR'),\n    },\n    'datalad.tests.temp.keep': {\n        'ui': ('yesno', {\n               'title': 'Function rmtemp will not remove temporary file/directory created for testing if this flag is set'}),\n        'type': EnsureBool(),\n    },\n    'datalad.tests.temp.fs': {\n        'ui': ('question', {\n               'title': 'Specify the temporary file system to use as loop device for testing DATALAD_TESTS_TEMP_DIR creation'}),\n    },\n    'datalad.tests.temp.fssize': {\n        'ui': ('question', {\n               'title': 'Specify the size of temporary file system to use as loop device for testing DATALAD_TESTS_TEMP_DIR creation'}),\n    },\n    'datalad.tests.ui.backend': {\n        'ui': ('question', {\n            'title': 'Tests UI backend',\n            # XXX we could add choices...\n            'text': 'Which UI backend to use'}),\n        'default': 'tests-noninteractive',\n    },\n    'datalad.tests.usecassette': {\n        'ui': ('question', {\n               'title': 'Specifies the location of the file to record network transactions by the VCR module. Currently used by when testing custom special remotes'}),\n    },\n    'datalad.tests.cache': {\n        'ui': ('question', {\n            'title': 'Cache directory for tests',\n            'text': 'Where should datalad cache test files?'}),\n        'destination': 'global',\n        'default_fn': lambda: opj(dirs.user_cache_dir, 'tests')\n    },\n    'datalad.log.level': {\n        'ui': ('question', {\n            'title': 'Used for control the verbosity of logs printed to '\n                     'stdout while running datalad commands/debugging'}),\n    },\n    'datalad.log.result-level': {\n        'ui': ('question', {\n               'title': 'Log level for command result messages',\n               'text': \"Overrides the default behavior of logging 'impossible' \"\n                       \"results as a warning, 'error' results as errors, and \"\n                       \"everything else as 'debug' with a single alternative \"\n                       \"log level\"}),\n        'type': EnsureChoice('debug', 'info', 'warning', 'error'),\n        # None keeps the default behavior\n        'default': 'None',\n    },\n    'datalad.log.name': {\n        'ui': ('question', {\n            'title': 'Include name of the log target in the log line'}),\n    },\n    'datalad.log.names': {\n        'ui': ('question', {\n            'title': 'Which names (,-separated) to print log lines for'}),\n    },\n    'datalad.log.namesre': {\n        'ui': ('question', {\n            'title': 'Regular expression for which names to print log lines for'}),\n    },\n    'datalad.log.outputs': {\n        'ui': ('question', {\n               'title': 'Whether to log stdout and stderr for executed commands',\n               'text': 'When enabled, setting the log level to 5 '\n                       'should catch all execution output, '\n                       'though some output may be logged at higher levels'}),\n        'default': False,\n        'type': EnsureBool(),\n    },\n    'datalad.log.timestamp': {\n        'ui': ('yesno', {\n               'title': 'Used to add timestamp to datalad logs'}),\n        'default': False,\n        'type': EnsureBool(),\n    },\n    'datalad.log.traceback': {\n        'ui': ('question', {\n               'title': 'Runs TraceBack function with collide set to True, if this flag is set to \"collide\". This replaces any common prefix between current traceback log and previous invocation with \"...\"'}),\n    },\n    'datalad.log.exc': {\n        'ui': ('yesno', {\n               'title': 'Include exceptions and their traceback in log messages. If set, \\'datalad.exc.str.tblimit\\' applies.'}),\n        'default': False,\n        'type': EnsureBool(),\n    },\n    'datalad.ssh.identityfile': {\n        'ui': ('question', {\n               'title': \"If set, pass this file as ssh's -i option.\"}),\n        'destination': 'global',\n        'default': None,\n    },\n    'datalad.ssh.multiplex-connections': {\n        'ui': ('question', {\n               'title': \"Whether to use a single shared connection for multiple SSH processes aiming at the same target.\"}),\n        'destination': 'global',\n        'default': not on_windows,\n        'type': EnsureBool(),\n    },\n    'datalad.annex.retry': {\n        'ui': ('question',\n               {'title': 'Value for annex.retry to use for git-annex calls',\n                'text': 'On transfer failure, annex.retry (sans \"datalad.\") '\n                        'controls the number of times that git-annex retries. '\n                        'DataLad will call git-annex with annex.retry set '\n                        'to the value here unless the annex.retry '\n                        'is explicitly configured'}),\n        'type': EnsureInt(),\n        'default': 3,\n    },\n    'datalad.repo.backend': {\n        'ui': ('question', {\n               'title': 'git-annex backend',\n               'text': 'Backend to use when creating git-annex repositories'}),\n        'default': 'MD5E',\n    },\n    'datalad.repo.direct': {\n        'ui': ('yesno', {\n               'title': 'Direct Mode for git-annex repositories',\n               'text': 'Set this flag to create annex repositories in direct mode by default'}),\n        'type': EnsureBool(),\n        'default': False,\n    },\n    'datalad.repo.version': {\n        'ui': ('question', {\n               'title': 'git-annex repository version',\n               'text': 'Specifies the repository version for git-annex to be used by default'}),\n        'type': EnsureInt(),\n        'default': 8,\n    },\n    'datalad.metadata.maxfieldsize': {\n        'ui': ('question', {\n               'title': 'Maximum metadata field size',\n               'text': 'Metadata fields exceeding this size (in bytes/chars) are excluded from metadata extractio'}),\n        'default': 100000,\n        'type': EnsureInt(),\n    },\n    'datalad.metadata.nativetype': {\n        'ui': ('question', {\n               'title': 'Native dataset metadata scheme',\n               'text': 'Set this label to engage a particular metadata extraction parser'}),\n    },\n    'datalad.metadata.store-aggregate-content': {\n        'ui': ('question', {\n               'title': 'Aggregated content metadata storage',\n               'text': 'If this flag is enabled, content metadata is aggregated into superdataset to allow for discovery of individual files. If disable unique content metadata values are still aggregated to enable dataset discovery'}),\n        'type': EnsureBool(),\n        'default': True,\n    },\n    'datalad.search.default-mode': {\n        'ui': ('question', {\n               'title': 'Default search mode',\n               'text': 'Label of the mode to be used by default'}),\n        'type': EnsureChoice('egrep', 'textblob', 'autofield'),  # graph,...\n        'default': 'egrep',\n    },\n    'datalad.search.index-default-documenttype': {\n        'ui': ('question', {\n               'title': 'Type of search index documents',\n               'text': 'Labels of document types to include in a default search index'}),\n        'type': EnsureChoice('all', 'datasets', 'files'),\n        'default': 'datasets',\n    },\n    'datalad.metadata.create-aggregate-annex-limit': {\n        'ui': ('question', {\n               'title': 'Limit configuration annexing aggregated metadata in new dataset',\n               'text': 'Git-annex large files expression (see https://git-annex.branchable.com/tips/largefiles; given expression will be wrapped in parentheses)'}),\n        'default': 'anything',\n    },\n    'datalad.runtime.max-annex-jobs': {\n        'ui': ('question', {\n               'title': 'Maximum number of git-annex jobs to request when \"jobs\" option set to \"auto\" (default)',\n               'text': 'Set this value to enable parallel annex jobs that may speed up certain operations (e.g. get file content). The effective number of jobs will not exceed the number of available CPU cores (or 3 if there is less than 3 cores).'}),\n        'type': EnsureInt(),\n        'default': 1,\n    },\n    'datalad.runtime.max-jobs': {\n        'ui': ('question', {\n            'title': 'Maximum number of jobs DataLad can run in \"parallel\"',\n            'text': 'Set this value to enable parallel multi-threaded DataLad jobs that may speed up certain '\n                    'operations, in particular operation across multiple datasets (e.g., install multiple '\n                    'subdatasets, etc).'}),\n        'type': EnsureInt(),\n        'default': 1,\n    },\n    'datalad.runtime.raiseonerror': {\n        'ui': ('question', {\n               'title': 'Error behavior',\n               'text': 'Set this flag to cause DataLad to raise an exception on errors that would have otherwise just get logged'}),\n        'type': EnsureBool(),\n        'default': False,\n    },\n    'datalad.runtime.report-status': {\n        'ui': ('question', {\n               'title': 'Command line result reporting behavior',\n               'text': \"If set (to other than 'all'), constrains command result report to records matching the given status. 'success' is a synonym for 'ok' OR 'notneeded', 'failure' stands for 'impossible' OR 'error'\"}),\n        'type': EnsureChoice('all', 'success', 'failure', 'ok', 'notneeded', 'impossible', 'error'),\n        'default': None,\n    },\n    'datalad.runtime.stalled-external': {\n        'ui': ('question', {\n            'title': 'Behavior for handing external processes',\n            'text': 'What to do with external processes if they do not finish in some minimal reasonable time. '\n                    'If \"abandon\", datalad would proceed without waiting for external process to exit. '\n                    'ATM applies only to batched git-annex processes. Should be changed with caution.'}),\n        'type': EnsureChoice('wait', 'abandon'),\n        'default': 'wait',\n    },\n    'datalad.search.indexercachesize': {\n        'ui': ('question', {\n               'title': 'Maximum cache size for search index (per process)',\n               'text': 'Actual memory consumption can be twice as high as this value in MB (one process per CPU is used)'}),\n        'default': 256,\n        'type': EnsureInt(),\n    },\n    'datalad.ui.progressbar': {\n        'ui': ('question', {\n            'title': 'UI progress bars',\n            'text': 'Default backend for progress reporting'}),\n        'default': None,\n        'type': EnsureChoice('tqdm', 'tqdm-ipython', 'log', 'none'),\n    },\n    'datalad.ui.color': {\n        'ui': ('question', {\n            'title': 'Colored terminal output',\n            'text': 'Enable or disable ANSI color codes in outputs; \"on\" overrides NO_COLOR environment variable'}),\n        'default': 'auto',\n        'type': EnsureChoice('on', 'off', 'auto'),\n    },\n    'datalad.ui.suppress-similar-results': {\n        'ui': ('question', {\n            'title': 'Suppress rendering of similar repetitive results',\n            'text': \"If enabled, after a certain number of subsequent \"\n                    \"results that are identical regarding key properties, \"\n                    \"such as 'status', 'action', and 'type', additional \"\n                    \"similar results are not rendered by the common result \"\n                    \"renderer anymore. Instead, a count \"\n                    \"of suppressed results is displayed. If disabled, or \"\n                    \"when not running in an interactive terminal, all results \"\n                    \"are rendered.\"}),\n        'default': True,\n        'type': EnsureBool(),\n    },\n    'datalad.ui.suppress-similar-results-threshold': {\n        'ui': ('question', {\n            'title': 'Threshold for suppressing similar repetitive results',\n            'text': \"Minimum number of similar results to occur before \"\n                    \"suppression is considered. \"\n                    \"See 'datalad.ui.suppress-similar-results' for more \"\n                    \"information.\"}),\n        'default': 10,\n        'type': EnsureInt(),\n    },\n    'datalad.save.no-message': {\n        'ui': ('question', {\n            'title': 'Commit message handling',\n            'text': 'When no commit message was provided: '\n                    'attempt to obtain one interactively (interactive); '\n                    'or use a generic commit message (generic). '\n                    'NOTE: The interactive option is experimental. The '\n                    'behavior may change in backwards-incompatible ways.'}),\n        'default': 'generic',\n        'type': EnsureChoice('interactive', 'generic'),\n    },\n    'datalad.install.inherit-local-origin': {\n        'ui': ('question', {\n            'title': 'Inherit local origin of dataset source',\n            'text': \"If enabled, a local 'origin' remote of a local dataset \"\n                    \"clone source is configured as an 'origin-2' remote \"\n                    \"to make its annex automatically available. The process \"\n                    \"is repeated recursively for any further qualifying \"\n                    \"'origin' dataset thereof.\"\n                    \"Note that if clone.defaultRemoteName is configured \"\n                    \"to use a name other than 'origin', that name will be \"\n                    \"used instead.\"}),\n        'default': True,\n        'type': EnsureBool(),\n    },\n}\n\n\ndef compute_cfg_defaults():\n    \"\"\"Compute dynamic defaults for configuration options.\n\n    These are options that depend on things like $HOME that change under our\n    testing setup.\n    \"\"\"\n    for key, value in definitions.items():\n        def_fn = value.get(\"default_fn\")\n        if def_fn:\n            value['default'] = def_fn()\n\n\ncompute_cfg_defaults()\n"}
A user will ask you to solve a task. You should first draft your thinking process (inner monologue). Then, generate the solution.

Your response format must follow the template below:
<think>
Your thoughts or/and draft, like working through an exercise on scratch paper. Be as casual and as long as you want until you are confident to generate a correct solution.
</think>
<solution>
Final solution presented to the user.
</solution>
system
We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
[ISSUE]

**Title:** Forced Credential Re-entry Not Triggered with `datalad.credentials.force-ask` Configuration

**Description:**

When attempting to update stored credentials by enabling the `datalad.credentials.force-ask` configuration setting, DataLad does not prompt for credential re-entry. This prevents users from updating their credentials seamlessly, as the existing credentials remain unchanged.

**Example Code:**

```python
from datalad.downloaders.credentials import UserPassword
from datalad import cfg as dlcfg

# Initialize keyring and set initial credentials
keyring = MemoryKeyring()
cred = UserPassword("name", keyring=keyring)
keyring.set('name', 'user', 'user1')
keyring.set('name', 'password', 'password1')

# Retrieve existing credentials
credentials = cred()
print(credentials)  # Outputs: {'user': 'user1', 'password': 'password1'}

# Attempt to force credential re-entry
dlcfg.set('datalad.credentials.force-ask', True)
credentials = cred()
print(credentials)  # Expected to prompt for new credentials and update
```

**Expected Behavior:**

When `datalad.credentials.force-ask` is set to `True`, calling `cred()` should prompt the user to re-enter their credentials. The new credentials provided by the user should then replace the existing ones in the keyring.

**Actual Behavior:**

Despite setting `force-ask` to `True`, calling `cred()` does not prompt for re-entry. The original credentials `{'user': 'user1', 'password': 'password1'}` are returned without any update, making it impossible to update the stored credentials through this configuration.

[/ISSUE]
--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.

--- BEGIN FILE ---
```
### datalad/interface/common_cfg.py
# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-
# ex: set sts=4 ts=4 sw=4 noet:
# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
#
#   See COPYING file distributed along with the datalad package for the
#   copyright and license terms.
#
# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
"""Common configuration options

"""

__docformat__ = 'restructuredtext'

from appdirs import AppDirs
from os import environ
from os.path import join as opj, expanduser
from datalad.support.constraints import EnsureBool
from datalad.support.constraints import EnsureInt
from datalad.support.constraints import EnsureNone
from datalad.support.constraints import EnsureChoice
from datalad.support.constraints import EnsureListOf
from datalad.support.constraints import EnsureStr
from datalad.utils import on_windows

dirs = AppDirs("datalad", "datalad.org")

subst_rule_docs = """\
A substitution specification is a string with a match and substitution
expression, each following Python's regular expression syntax. Both expressions
are concatenated to a single string with an arbitrary delimiter character. The
delimiter is defined by prefixing the string with the delimiter. Prefix and
delimiter are stripped from the expressions (Example:
",^http://(.*)$,https://\\1").  This setting can be defined multiple times.
Substitutions will be applied incrementally, in order of their definition. The
first substitution in such a series must match, otherwise no further
substitutions in a series will be considered. However, following the first
match all further substitutions in a series are processed, regardless whether
intermediate expressions match or not."""

definitions = {
    'datalad.clone.url-substitute.github': {
        'ui': ('question', {
               'title': 'GitHub URL substitution rule',
               'text': 'Mangling for GitHub-related URL. ' + subst_rule_docs
        }),
        'destination': 'global',
        'default': (
            # take any github project URL apart into <org>###<identifier>
            r',https?://github.com/([^/]+)/(.*)$,\1###\2',
            # replace any (back)slashes with a single dash
            r',[/\\]+,-',
            # replace any whitespace (include urlquoted variant)
            # with a single underscore
            r',\s+|(%2520)+|(%20)+,_',
            # rebuild functional project URL
            r',([^#]+)###(.*),https://github.com/\1/\2',
        )
    },
    # TODO this one should migrate to the datalad-osf extension. however, right
    # now extensions cannot provide default configuration
    # https://github.com/datalad/datalad/issues/5769
    'datalad.clone.url-substitute.osf': {
        'ui': ('question', {
               'title': 'Open Science Framework URL substitution rule',
               'text': 'Mangling for OSF-related URLs. ' + subst_rule_docs
        }),
        'destination': 'global',
        'default': (
            # accept browser-provided URL and convert to those accepted by
            # the datalad-osf extension
            r',^https://osf.io/([^/]+)[/]*$,osf://\1',
        )
    },
    # this is actually used in downloaders, but kept cfg name original
    'datalad.crawl.cache': {
        'ui': ('yesno', {
               'title': 'Crawler download caching',
               'text': 'Should the crawler cache downloaded files?'}),
        'destination': 'local',
        'type': bool,
    },
    'datalad.externals.nda.dbserver': {
        'ui': ('question', {
               'title': 'NDA database server',
               'text': 'Hostname of the database server'}),
        'destination': 'global',
        # Development one is https://development.nimhda.org
        'default': 'https://nda.nih.gov/DataManager/dataManager',
    },
    'datalad.locations.cache': {
        'ui': ('question', {
               'title': 'Cache directory',
               'text': 'Where should datalad cache files?'}),
        'destination': 'global',
        'default_fn': lambda: dirs.user_cache_dir,
    },
    'datalad.locations.default-dataset': {
        'ui': ('question', {
               'title': 'Default dataset path',
               'text': 'Where should datalad should look for (or install) a '
                       'default dataset?'}),
        'destination': 'global',
        'default_fn': lambda: opj(expanduser('~'), 'datalad'),
    },
    'datalad.locations.locks': {
        'ui': ('question', {
               'title': 'Lockfile directory',
               'text': 'Where should datalad store lock files?'}),
        'destination': 'global',
        'default_fn': lambda: opj(dirs.user_cache_dir, 'locks')
    },
    'datalad.locations.sockets': {
        'ui': ('question', {
               'title': 'Socket directory',
               'text': 'Where should datalad store socket files?'}),
        'destination': 'global',
        'default_fn': lambda: opj(dirs.user_cache_dir, 'sockets'),
    },
    'datalad.locations.system-procedures': {
        'ui': ('question', {
               'title': 'System procedure directory',
               'text': 'Where should datalad search for system procedures?'}),
        'destination': 'global',
        'default_fn': lambda: opj(dirs.site_config_dir, 'procedures'),
    },
    'datalad.locations.user-procedures': {
        'ui': ('question', {
               'title': 'User procedure directory',
               'text': 'Where should datalad search for user procedures?'}),
        'destination': 'global',
        'default_fn': lambda: opj(dirs.user_config_dir, 'procedures'),
    },
    'datalad.locations.extra-procedures': {
        'ui': ('question', {
            'title': 'Extra procedure directory',
            'text': 'Where should datalad search for some additional procedures?'}),
        'destination': 'global',
    },
    'datalad.locations.dataset-procedures': {
        'ui': ('question', {
               'title': 'Dataset procedure directory',
               'text': 'Where should datalad search for dataset procedures (relative to a dataset root)?'}),
        'destination': 'dataset',
        'default': opj('.datalad', 'procedures'),
    },
    'datalad.exc.str.tblimit': {
        'ui': ('question', {
               'title': 'This flag is used by datalad to cap the number of traceback steps included in exception logging and result reporting to DATALAD_EXC_STR_TBLIMIT of pre-processed entries from traceback.'}),
    },
    'datalad.fake-dates': {
        'ui': ('yesno', {
               'title': 'Fake (anonymize) dates',
               'text': 'Should the dates in the logs be faked?'}),
        'destination': 'local',
        'type': EnsureBool(),
        'default': False,
    },
    'datalad.fake-dates-start': {
        'ui': ('question', {
            'title': 'Initial fake date',
            'text': 'When faking dates and there are no commits in any local branches, generate the date by adding one second to this value (Unix epoch time). The value must be positive.'}),
        'type': EnsureInt(),
        'default': 1112911993,
    },
    'datalad.github.token-note': {
        'ui': ('question', {
            'title': 'Github token note',
            'text': 'Description for a Personal access token to generate.'}),
        'default': 'DataLad',
    },
    'datalad.tests.nonetwork': {
        'ui': ('yesno', {
               'title': 'Skips network tests completely if this flag is set Examples include test for s3, git_repositories, openfmri etc'}),
        'type': EnsureBool(),
    },
    'datalad.tests.nonlo': {
        'ui': ('question', {
               'title': 'Specifies network interfaces to bring down/up for testing. Currently used by travis.'}),
    },
    'datalad.tests.noteardown': {
        'ui': ('yesno', {
               'title': 'Does not execute teardown_package which cleans up temp files and directories created by tests if this flag is set'}),
        'type': EnsureBool(),
    },
    'datalad.tests.dataladremote': {
        'ui': ('yesno', {
               'title': 'Binary flag to specify whether each annex repository should get datalad special remote in every test repository'}),
        'type': EnsureBool(),
    },
    'datalad.tests.runcmdline': {
        'ui': ('yesno', {
               'title': 'Binary flag to specify if shell testing using shunit2 to be carried out'}),
        'type': EnsureBool(),
    },
    'datalad.tests.ssh': {
        'ui': ('yesno', {
               'title': 'Skips SSH tests if this flag is **not** set'}),
        'type': EnsureBool(),
    },
    'datalad.tests.knownfailures.skip': {
        'ui': ('yesno', {
               'title': 'Skips tests that are known to currently fail'}),
        'type': EnsureBool(),
        'default': True,
    },
    'datalad.tests.knownfailures.probe': {
        'ui': ('yesno', {
               'title': 'Probes tests that are known to fail on whether or not they are actually still failing'}),
        'type': EnsureBool(),
        'default': False,
    },
    'datalad.tests.setup.testrepos': {
        'ui': ('question', {
            'title': 'Pre-creates repositories for @with_testrepos within setup_package'}),
        'type': EnsureBool(),
        'default': False,
    },
    'datalad.tests.temp.dir': {
        'ui': ('question', {
               'title': 'Create a temporary directory at location specified by this flag. It is used by tests to create a temporary git directory while testing git annex archives etc'}),
        'type': EnsureStr(),
        'default_fn': lambda: environ.get('TMPDIR'),
    },
    'datalad.tests.temp.keep': {
        'ui': ('yesno', {
               'title': 'Function rmtemp will not remove temporary file/directory created for testing if this flag is set'}),
        'type': EnsureBool(),
    },
    'datalad.tests.temp.fs': {
        'ui': ('question', {
               'title': 'Specify the temporary file system to use as loop device for testing DATALAD_TESTS_TEMP_DIR creation'}),
    },
    'datalad.tests.temp.fssize': {
        'ui': ('question', {
               'title': 'Specify the size of temporary file system to use as loop device for testing DATALAD_TESTS_TEMP_DIR creation'}),
    },
    'datalad.tests.ui.backend': {
        'ui': ('question', {
            'title': 'Tests UI backend',
            # XXX we could add choices...
            'text': 'Which UI backend to use'}),
        'default': 'tests-noninteractive',
    },
    'datalad.tests.usecassette': {
        'ui': ('question', {
               'title': 'Specifies the location of the file to record network transactions by the VCR module. Currently used by when testing custom special remotes'}),
    },
    'datalad.tests.cache': {
        'ui': ('question', {
            'title': 'Cache directory for tests',
            'text': 'Where should datalad cache test files?'}),
        'destination': 'global',
        'default_fn': lambda: opj(dirs.user_cache_dir, 'tests')
    },
    'datalad.log.level': {
        'ui': ('question', {
            'title': 'Used for control the verbosity of logs printed to '
                     'stdout while running datalad commands/debugging'}),
    },
    'datalad.log.result-level': {
        'ui': ('question', {
               'title': 'Log level for command result messages',
               'text': "Overrides the default behavior of logging 'impossible' "
                       "results as a warning, 'error' results as errors, and "
                       "everything else as 'debug' with a single alternative "
                       "log level"}),
        'type': EnsureChoice('debug', 'info', 'warning', 'error'),
        # None keeps the default behavior
        'default': 'None',
    },
    'datalad.log.name': {
        'ui': ('question', {
            'title': 'Include name of the log target in the log line'}),
    },
    'datalad.log.names': {
        'ui': ('question', {
            'title': 'Which names (,-separated) to print log lines for'}),
    },
    'datalad.log.namesre': {
        'ui': ('question', {
            'title': 'Regular expression for which names to print log lines for'}),
    },
    'datalad.log.outputs': {
        'ui': ('question', {
               'title': 'Whether to log stdout and stderr for executed commands',
               'text': 'When enabled, setting the log level to 5 '
                       'should catch all execution output, '
                       'though some output may be logged at higher levels'}),
        'default': False,
        'type': EnsureBool(),
    },
    'datalad.log.timestamp': {
        'ui': ('yesno', {
               'title': 'Used to add timestamp to datalad logs'}),
        'default': False,
        'type': EnsureBool(),
    },
    'datalad.log.traceback': {
        'ui': ('question', {
               'title': 'Runs TraceBack function with collide set to True, if this flag is set to "collide". This replaces any common prefix between current traceback log and previous invocation with "..."'}),
    },
    'datalad.log.exc': {
        'ui': ('yesno', {
               'title': 'Include exceptions and their traceback in log messages. If set, \'datalad.exc.str.tblimit\' applies.'}),
        'default': False,
        'type': EnsureBool(),
    },
    'datalad.ssh.identityfile': {
        'ui': ('question', {
               'title': "If set, pass this file as ssh's -i option."}),
        'destination': 'global',
        'default': None,
    },
    'datalad.ssh.multiplex-connections': {
        'ui': ('question', {
               'title': "Whether to use a single shared connection for multiple SSH processes aiming at the same target."}),
        'destination': 'global',
        'default': not on_windows,
        'type': EnsureBool(),
    },
    'datalad.annex.retry': {
        'ui': ('question',
               {'title': 'Value for annex.retry to use for git-annex calls',
                'text': 'On transfer failure, annex.retry (sans "datalad.") '
                        'controls the number of times that git-annex retries. '
                        'DataLad will call git-annex with annex.retry set '
                        'to the value here unless the annex.retry '
                        'is explicitly configured'}),
        'type': EnsureInt(),
        'default': 3,
    },
    'datalad.repo.backend': {
        'ui': ('question', {
               'title': 'git-annex backend',
               'text': 'Backend to use when creating git-annex repositories'}),
        'default': 'MD5E',
    },
    'datalad.repo.direct': {
        'ui': ('yesno', {
               'title': 'Direct Mode for git-annex repositories',
               'text': 'Set this flag to create annex repositories in direct mode by default'}),
        'type': EnsureBool(),
        'default': False,
    },
    'datalad.repo.version': {
        'ui': ('question', {
               'title': 'git-annex repository version',
               'text': 'Specifies the repository version for git-annex to be used by default'}),
        'type': EnsureInt(),
        'default': 8,
    },
    'datalad.metadata.maxfieldsize': {
        'ui': ('question', {
               'title': 'Maximum metadata field size',
               'text': 'Metadata fields exceeding this size (in bytes/chars) are excluded from metadata extractio'}),
        'default': 100000,
        'type': EnsureInt(),
    },
    'datalad.metadata.nativetype': {
        'ui': ('question', {
               'title': 'Native dataset metadata scheme',
               'text': 'Set this label to engage a particular metadata extraction parser'}),
    },
    'datalad.metadata.store-aggregate-content': {
        'ui': ('question', {
               'title': 'Aggregated content metadata storage',
               'text': 'If this flag is enabled, content metadata is aggregated into superdataset to allow for discovery of individual files. If disable unique content metadata values are still aggregated to enable dataset discovery'}),
        'type': EnsureBool(),
        'default': True,
    },
    'datalad.search.default-mode': {
        'ui': ('question', {
               'title': 'Default search mode',
               'text': 'Label of the mode to be used by default'}),
        'type': EnsureChoice('egrep', 'textblob', 'autofield'),  # graph,...
        'default': 'egrep',
    },
    'datalad.search.index-default-documenttype': {
        'ui': ('question', {
               'title': 'Type of search index documents',
               'text': 'Labels of document types to include in a default search index'}),
        'type': EnsureChoice('all', 'datasets', 'files'),
        'default': 'datasets',
    },
    'datalad.metadata.create-aggregate-annex-limit': {
        'ui': ('question', {
               'title': 'Limit configuration annexing aggregated metadata in new dataset',
               'text': 'Git-annex large files expression (see https://git-annex.branchable.com/tips/largefiles; given expression will be wrapped in parentheses)'}),
        'default': 'anything',
    },
    'datalad.runtime.max-annex-jobs': {
        'ui': ('question', {
               'title': 'Maximum number of git-annex jobs to request when "jobs" option set to "auto" (default)',
               'text': 'Set this value to enable parallel annex jobs that may speed up certain operations (e.g. get file content). The effective number of jobs will not exceed the number of available CPU cores (or 3 if there is less than 3 cores).'}),
        'type': EnsureInt(),
        'default': 1,
    },
    'datalad.runtime.max-jobs': {
        'ui': ('question', {
            'title': 'Maximum number of jobs DataLad can run in "parallel"',
            'text': 'Set this value to enable parallel multi-threaded DataLad jobs that may speed up certain '
                    'operations, in particular operation across multiple datasets (e.g., install multiple '
                    'subdatasets, etc).'}),
        'type': EnsureInt(),
        'default': 1,
    },
    'datalad.runtime.raiseonerror': {
        'ui': ('question', {
               'title': 'Error behavior',
               'text': 'Set this flag to cause DataLad to raise an exception on errors that would have otherwise just get logged'}),
        'type': EnsureBool(),
        'default': False,
    },
    'datalad.runtime.report-status': {
        'ui': ('question', {
               'title': 'Command line result reporting behavior',
               'text': "If set (to other than 'all'), constrains command result report to records matching the given status. 'success' is a synonym for 'ok' OR 'notneeded', 'failure' stands for 'impossible' OR 'error'"}),
        'type': EnsureChoice('all', 'success', 'failure', 'ok', 'notneeded', 'impossible', 'error'),
        'default': None,
    },
    'datalad.runtime.stalled-external': {
        'ui': ('question', {
            'title': 'Behavior for handing external processes',
            'text': 'What to do with external processes if they do not finish in some minimal reasonable time. '
                    'If "abandon", datalad would proceed without waiting for external process to exit. '
                    'ATM applies only to batched git-annex processes. Should be changed with caution.'}),
        'type': EnsureChoice('wait', 'abandon'),
        'default': 'wait',
    },
    'datalad.search.indexercachesize': {
        'ui': ('question', {
               'title': 'Maximum cache size for search index (per process)',
               'text': 'Actual memory consumption can be twice as high as this value in MB (one process per CPU is used)'}),
        'default': 256,
        'type': EnsureInt(),
    },
    'datalad.ui.progressbar': {
        'ui': ('question', {
            'title': 'UI progress bars',
            'text': 'Default backend for progress reporting'}),
        'default': None,
        'type': EnsureChoice('tqdm', 'tqdm-ipython', 'log', 'none'),
    },
    'datalad.ui.color': {
        'ui': ('question', {
            'title': 'Colored terminal output',
            'text': 'Enable or disable ANSI color codes in outputs; "on" overrides NO_COLOR environment variable'}),
        'default': 'auto',
        'type': EnsureChoice('on', 'off', 'auto'),
    },
    'datalad.ui.suppress-similar-results': {
        'ui': ('question', {
            'title': 'Suppress rendering of similar repetitive results',
            'text': "If enabled, after a certain number of subsequent "
                    "results that are identical regarding key properties, "
                    "such as 'status', 'action', and 'type', additional "
                    "similar results are not rendered by the common result "
                    "renderer anymore. Instead, a count "
                    "of suppressed results is displayed. If disabled, or "
                    "when not running in an interactive terminal, all results "
                    "are rendered."}),
        'default': True,
        'type': EnsureBool(),
    },
    'datalad.ui.suppress-similar-results-threshold': {
        'ui': ('question', {
            'title': 'Threshold for suppressing similar repetitive results',
            'text': "Minimum number of similar results to occur before "
                    "suppression is considered. "
                    "See 'datalad.ui.suppress-similar-results' for more "
                    "information."}),
        'default': 10,
        'type': EnsureInt(),
    },
    'datalad.save.no-message': {
        'ui': ('question', {
            'title': 'Commit message handling',
            'text': 'When no commit message was provided: '
                    'attempt to obtain one interactively (interactive); '
                    'or use a generic commit message (generic). '
                    'NOTE: The interactive option is experimental. The '
                    'behavior may change in backwards-incompatible ways.'}),
        'default': 'generic',
        'type': EnsureChoice('interactive', 'generic'),
    },
    'datalad.install.inherit-local-origin': {
        'ui': ('question', {
            'title': 'Inherit local origin of dataset source',
            'text': "If enabled, a local 'origin' remote of a local dataset "
                    "clone source is configured as an 'origin-2' remote "
                    "to make its annex automatically available. The process "
                    "is repeated recursively for any further qualifying "
                    "'origin' dataset thereof."
                    "Note that if clone.defaultRemoteName is configured "
                    "to use a name other than 'origin', that name will be "
                    "used instead."}),
        'default': True,
        'type': EnsureBool(),
    },
}


def compute_cfg_defaults():
    """Compute dynamic defaults for configuration options.

    These are options that depend on things like $HOME that change under our
    testing setup.
    """
    for key, value in definitions.items():
        def_fn = value.get("default_fn")
        if def_fn:
            value['default'] = def_fn()


compute_cfg_defaults()


### datalad/downloaders/credentials.py
# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-
# ex: set sts=4 ts=4 sw=4 noet:
# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
#
#   See COPYING file distributed along with the datalad package for the
#   copyright and license terms.
#
# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
"""Interface information about credentials

Provides minimalistic interface to deal (query, request, store) with most common
types of credentials.  To be used by Authenticators
"""

__dev_doc__ = """
Possibly useful in the future 3rd part developments

https://github.com/omab/python-social-auth
    social authentication/registration mechanism with support for several
    frameworks and auth providers
"""

import time

from collections import OrderedDict

from ..dochelpers import exc_str
from ..support.exceptions import AccessDeniedError
from ..support.keyring_ import keyring as keyring_
from ..ui import ui
from ..utils import auto_repr
from ..support.network import iso8601_to_epoch

from datalad import cfg as dlcfg

from logging import getLogger
lgr = getLogger('datalad.downloaders.credentials')


@auto_repr
class Credential(object):
    """Base class for different types of credentials
    """

    # Should be defined in a subclass as an OrderedDict of fields
    # name -> dict(attributes)
    _FIELDS = None
    _KNOWN_ATTRS = {
        'hidden',    # UI should not display the value
        'repeat',    # UI should repeat entry or not. Set to False to override default logic
        'optional',  # Not mandatory thus not requested if not set
    }

    def __init__(self, name, url=None, keyring=None):
        """
        Parameters
        ----------
        name : str
            Name of the credential, as it would be identified by in the centralized
            storage of credentials
        url : str, optional
            URL string to point users to where to seek obtaining the credentials
        keyring : a keyring
            An object providing (g|s)et_password.  If None, keyring module is used
            as is
        """
        self.name = name
        self.url = url
        self._keyring = keyring or keyring_
        self._prepare()

    def _prepare(self):
        """Additional house-keeping possibly to be performed in subclasses

        Created to avoid all the passing args/kwargs of __init__ all the time
        """
        # Basic checks
        for f, fattrs in self._FIELDS.items():
            unknown_attrs = set(fattrs).difference(self._KNOWN_ATTRS)
            if unknown_attrs:
                raise ValueError("Unknown attributes %s. Known are: %s"
                                 % (unknown_attrs, self._KNOWN_ATTRS))

    def _is_field_optional(self, f):
        return self._FIELDS[f].get('optional', False)

    @property
    def is_known(self):
        """Return True if values for all fields of the credential are known"""
        try:
            return all(
                self._is_field_optional(f) or self._get_field_value(f) is not None
                for f in self._FIELDS)
        except Exception as exc:
            lgr.warning("Failed to query keyring: %s" % exc_str(exc))
            return False

    def _get_field_value(self, field):
        return dlcfg.get('datalad.credential.{name}.{field}'.format(
            name=self.name,
            field=field.replace('_', '-')
        )) or self._keyring.get(self.name, field)

    def _ask_field_value(self, f, instructions=None):
        msg = instructions if instructions else \
            ("You need to authenticate with %r credentials." % self.name +
                  (" %s provides information on how to gain access"
                   % self.url if self.url else ''))

        # provide custom options only if set for the field
        f_props = self._FIELDS[f]
        kwargs = {}
        for p in ('hidden', 'repeat'):
            if p in f_props:
                kwargs[p] = f_props[p]
        return ui.question(
            f,
            title=msg,
            **kwargs
        )

    def _ask_and_set(self, f, instructions=None):
        v = self._ask_field_value(f, instructions=instructions)
        self.set(**{f: v})
        return v

    def enter_new(self, instructions=None, **kwargs):
        """Enter new values for the credential fields

        Parameters
        ----------
        instructions : str, optional
          If given, the auto-generated instructions based on a login-URL are
          replaced by the given string
        **kwargs
          Any given key value pairs with non-None values are used to set the
          field `key` to the given value, without asking for user input
        """
        if kwargs:
            unknown_fields = set(kwargs).difference(self._FIELDS)
            known_fields = set(self._FIELDS).difference(kwargs)
            if unknown_fields:
                raise ValueError(
                    "Unknown to %s field(s): %s.  Known but not specified: %s"
                    % (self,
                       ', '.join(sorted(unknown_fields)),
                       ', '.join(sorted(known_fields))
                       ))
        # Use ui., request credential fields corresponding to the type
        for f in self._FIELDS:
            if kwargs.get(f, None):
                # use given value, don't ask
                self.set(**{f: kwargs[f]})
            elif not self._is_field_optional(f):
                self._ask_and_set(f, instructions=instructions)

    def __call__(self):
        """Obtain credentials from a keyring and if any is not known -- ask"""
        fields = {}
        for f in self._FIELDS:
            v = self._get_field_value(f)
            if not self._is_field_optional(f):
                while v is None:  # was not known
                    v = self._ask_and_set(f)
                fields[f] = v
            elif v is not None:
                fields[f] = v
        return fields

    def set(self, **kwargs):
        """Set field(s) of the credential"""
        for f, v in kwargs.items():
            if f not in self._FIELDS:
                raise ValueError("Unknown field %s. Known are: %s"
                                 % (f, self._FIELDS.keys()))
            self._keyring.set(self.name, f, v)

    def get(self, f, default=None):
        """Get a field of the credential"""
        if f not in self._FIELDS:
            raise ValueError("Unknown field %s. Known are: %s"
                             % (f, self._FIELDS.keys()))
        try:
            return self._get_field_value(f)
        except:  # MIH: what could even happen? _keyring not a dict?
            return default

    def delete(self):
        """Deletes credential values from the keyring"""
        for f in self._FIELDS:
            self._keyring.delete(self.name, f)


class UserPassword(Credential):
    """Simple type of a credential which consists of user/password pair"""

    _FIELDS = OrderedDict([('user', {}),
                           ('password', {'hidden': True})])

    is_expired = False  # no expiration provisioned


class Token(Credential):
    """Simple type of a credential which provides a single token"""

    _FIELDS = OrderedDict([('token', {'hidden': True, 'repeat': False})])

    is_expired = False  # no expiration provisioned


class AWS_S3(Credential):
    """Credential for AWS S3 service"""

    _FIELDS = OrderedDict([('key_id', {'repeat': False}),
                           ('secret_id', {'hidden': True, 'repeat': False}),
                           ('session', {'optional': True}),
                           ('expiration', {'optional': True}),
                           ])

    @property
    def is_expired(self):
        exp = self.get('expiration', None)
        if not exp:
            return False
        exp_epoch = iso8601_to_epoch(exp)
        # -2 to artificially shorten duration of the allotment to avoid
        # possible race conditions between us checking either it has
        # already expired before submitting a request.
        expire_in = (exp_epoch - time.time() - 2) / 3600.

        lgr.debug(
            ("Credential %s has expired %.2fh ago"
                if expire_in <= 0 else "Credential %s will expire in %.2fh")
            % (self.name, expire_in))
        return expire_in <= 0


@auto_repr
class CompositeCredential(Credential):
    """Credential which represent a sequence of Credentials where front one is exposed to user
    """

    # To be defined in sub-classes
    _CREDENTIAL_CLASSES = None
    _CREDENTIAL_ADAPTERS = None

    def _prepare(self):
        assert len(self._CREDENTIAL_CLASSES) > 1, "makes sense only if there is > 1 credential"
        assert len(self._CREDENTIAL_CLASSES) == len(self._CREDENTIAL_ADAPTERS) + 1, \
            "there should be 1 less of adapter than _CREDENTIAL_CLASSES"

        for C in self._CREDENTIAL_CLASSES:
            assert issubclass(C, Credential), "%s must be a subclass of Credential" % C

        # First credential should bear the name and url
        credentials = [self._CREDENTIAL_CLASSES[0](self.name, url=self.url, keyring=self._keyring)]
        # and we just reuse its _FIELDS for _ask_field_value etc
        self._FIELDS = credentials[0]._FIELDS
        # the rest with index suffix, but storing themselves in the same keyring
        for iC, C in enumerate(self._CREDENTIAL_CLASSES[1:]):
            credentials.append(
                C(name="%s:%d" % (self.name, iC + 1), url=None, keyring=self._keyring)
            )
        self._credentials = credentials

        super(CompositeCredential, self)._prepare()

    # Here it becomes tricky, since theoretically it is the "tail"
    # ones which might expire etc, so we wouldn't exactly know what
    # new credentials outside process wanted -- it would be silly to ask
    # right away the "entry" credentials if it is just the expiration of the
    # tail credentials
    def enter_new(self):
        # should invalidate/remove all tail credentials to avoid failing attempts to login
        self._credentials[0].enter_new()
        self.refresh()

    def refresh(self):
        """Re-establish "dependent" credentials

        E.g. if code outside was reported that it expired somehow before known expiration datetime
        """
        for c in self._credentials[1:]:
            c.delete()
        # trigger re-establishing the chain
        _ = self()
        if self.is_expired:
            raise RuntimeError("Credential %s expired right upon refresh: should have not happened")

    @property
    def is_expired(self):
        return any(c.is_expired for c in self._credentials)

    def __call__(self):
        """Obtain credentials from a keyring and if any is not known -- ask"""
        # Start from the tail until we have credentials set
        idx = len(self._credentials) - 1
        for c in self._credentials[::-1]:
            if c.is_known and not c.is_expired:
                break
            idx -= 1

        if idx < 0:
            # none was known, all the same -- start with the first one
            idx = 0

        # TODO: consider moving logic of traversal into authenticator since it is
        # the one spitting out authentication error etc
        # Theoretically we could just reuse 'fields' from adapter in the next step
        # but let's do full circle, so that if any "normalization" is done by
        # Credentials we take that into account
        for c, adapter, next_c in zip(
                self._credentials[idx:],
                self._CREDENTIAL_ADAPTERS[idx:],
                self._credentials[idx + 1:]):
            fields = c()
            next_fields = adapter(self, **fields)
            next_c.set(**next_fields)

        return self._credentials[-1]()


def _nda_adapter(composite, user=None, password=None):
    from datalad.support.third.nda_aws_token_generator import NDATokenGenerator
    from .. import cfg
    nda_auth_url = cfg.obtain('datalad.externals.nda.dbserver')
    gen = NDATokenGenerator(nda_auth_url)
    lgr.debug("Generating token for NDA user %s using %s talking to %s",
              user, gen, nda_auth_url)
    try:
        token = gen.generate_token(user, password)
    except Exception as exc:  # it is really just an "Exception"
        exc_str = str(exc).lower()
        # ATM it is "Invalid username and/or password"
        # but who knows what future would bring
        if "invalid" in exc_str and ("user" in exc_str or "password" in exc_str):
            raise AccessDeniedError(exc_str)
        raise
    # There are also session and expiration we ignore... TODO anything about it?!!!
    # we could create a derived AWS_S3 which would also store session and expiration
    # and then may be Composite could use those????
    return dict(key_id=token.access_key, secret_id=token.secret_key,
                session=token.session, expiration=token.expiration)


class NDA_S3(CompositeCredential):
    """Credential to access NDA AWS

    So for NDA we need a credential which is a composite credential.
    User provides UserPassword and then some adapter generates AWS_S3
    out of it
    """
    _CREDENTIAL_CLASSES = (UserPassword, AWS_S3)
    _CREDENTIAL_ADAPTERS = (_nda_adapter,)


def _loris_adapter(composite, user=None, password=None, **kwargs):
    from datalad.support.third.loris_token_generator import LORISTokenGenerator

    gen = LORISTokenGenerator(url=composite.url)
    token = gen.generate_token(user, password)

    return dict(token=token)


class LORIS_Token(CompositeCredential):
    _CREDENTIAL_CLASSES = (UserPassword, Token)
    _CREDENTIAL_ADAPTERS = (_loris_adapter,)

    def __init__(self, name, url=None, keyring=None):
        super(CompositeCredential, self).__init__(name, url, keyring)


CREDENTIAL_TYPES = {
    'user_password': UserPassword,
    'aws-s3': AWS_S3,
    'nda-s3': NDA_S3,
    'token': Token,
    'loris-token': LORIS_Token
}

```
--- END FILE ---

Please first localize the bug based on the issue statement, and then generate *SEARCH/REPLACE* edits to fix the issue.

Every *SEARCH/REPLACE* edit must use this format:
1. The file path
2. The start of search block: <<<<<<< SEARCH
3. A contiguous chunk of lines to search for in the existing source code
4. The dividing line: =======
5. The lines to replace into the source code
6. The end of the replace block: >>>>>>> REPLACE

Here is an example:

```python
### mathweb/flask/app.py
<<<<<<< SEARCH
from flask import Flask
=======
import math
from flask import Flask
>>>>>>> REPLACE
```
</solution>

Please note that the *SEARCH/REPLACE* edit REQUIRES PROPER INDENTATION. If you would like to add the line '        print(x)', you must fully write that out, with all those spaces before the code!
Wrap each *SEARCH/REPLACE* edit in a code block as shown in the example above. If you have multiple *SEARCH/REPLACE* edits, use a separate code block for each one.
user