{
    "file_diffs": [
        {
            "old_file_content": "",
            "new_file_content": "",
            "header": {
                "file": {
                    "path": "datalad/metadata/indexers/__init__.py"
                },
                "misc_line": "new file mode 100644"
            },
            "index_line": {
                "old_commit_hash": "000000000",
                "new_commit_hash": "e69de29bb",
                "mode": ""
            },
            "is_binary_file": false,
            "binary_line": null,
            "minus_file": null,
            "plus_file": null,
            "hunks": []
        },
        {
            "old_file_content": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\" Metadata indexer base class \"\"\"\n\nfrom typing import Any, Dict, List, Union\n\n\nclass MetadataIndexer(object):\n    \"\"\" Defines the indexer interface \"\"\"\n    def __init__(self, metadata_format_name: str):\n        \"\"\"\n        Create a metadata indexer\n\n        The format name is passed to the constructor to allow\n        a single indexer to process multiple extractor results\n        \"\"\"\n        self.metadata_format_name = metadata_format_name\n\n    def create_index(self, metadata: Union[Dict, List]) -> Dict[str, Any]:\n        \"\"\"\n        Create an index from metadata.\n\n        The input is a list or dictionary that contains metadata\n        in the format identified by metadata_format_name.\n\n        The output should be a set of key-value pairs that represent\n        the information stored in `metadata´.\n\n        Parameters\n        ----------\n        metadata : Dict or List\n          Metadata created by an extractor.\n\n        Returns\n        ----------\n        dict:\n           key-value pairs representing the information in metadata.\n           values can be literals or lists of literals\n        \"\"\"\n        raise NotImplementedError\n",
            "new_file_content": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\" Metadata indexer base class \"\"\"\n\nfrom typing import Any, Dict, List, Union\n\n\nclass MetadataIndexer(object):\n    \"\"\" Defines the indexer interface \"\"\"\n    def __init__(self, metadata_format_name: str):\n        \"\"\"\n        Create a metadata indexer\n\n        The format name is passed to the constructor to allow\n        a single indexer to process multiple extractor results\n        \"\"\"\n        self.metadata_format_name = metadata_format_name\n\n    def create_index(self, metadata: Union[Dict, List]) -> Dict[str, Any]:\n        \"\"\"\n        Create an index from metadata.\n\n        The input is a list or dictionary that contains metadata\n        in the format identified by metadata_format_name.\n\n        The output should be a set of key-value pairs that represent\n        the information stored in `metadata´.\n\n        Parameters\n        ----------\n        metadata : Dict or List\n          Metadata created by an extractor.\n\n        Returns\n        -------\n        dict:\n           key-value pairs representing the information in metadata.\n           values can be literals or lists of literals\n        \"\"\"\n        raise NotImplementedError\n",
            "header": {
                "file": {
                    "path": "datalad/metadata/indexers/base.py"
                },
                "misc_line": null
            },
            "index_line": {
                "old_commit_hash": "0c015a3e0",
                "new_commit_hash": "27f3a416c",
                "mode": "100644"
            },
            "is_binary_file": false,
            "binary_line": null,
            "minus_file": {
                "path": "a/datalad/metadata/indexers/base.py"
            },
            "plus_file": {
                "path": "b/datalad/metadata/indexers/base.py"
            },
            "hunks": [
                {
                    "descriptor": {
                        "old_range": {
                            "start": 38,
                            "length": 7
                        },
                        "new_range": {
                            "start": 38,
                            "length": 7
                        },
                        "section": "class MetadataIndexer(object):"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "          Metadata created by an extractor.",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "        Returns",
                                "type": "context"
                            },
                            {
                                "content": "        ----------",
                                "type": "deleted"
                            },
                            {
                                "content": "        -------",
                                "type": "added"
                            },
                            {
                                "content": "        dict:",
                                "type": "context"
                            },
                            {
                                "content": "           key-value pairs representing the information in metadata.",
                                "type": "context"
                            },
                            {
                                "content": "           values can be literals or lists of literals",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [
                        {
                            "file_name": "datalad/metadata/indexers/base.py",
                            "type": "method",
                            "name": "MetadataIndexer.create_index",
                            "content": "class MetadataIndexer(object):\n    \"\"\" Defines the indexer interface \"\"\"\n    def __init__(self, metadata_format_name: str):\n        \"\"\"\n        Create a metadata indexer\n\n        The format name is passed to the constructor to allow\n        a single indexer to process multiple extractor results\n        \"\"\"\n        self.metadata_format_name = metadata_format_name\n\n    def create_index(self, metadata: Union[Dict, List]) -> Dict[str, Any]:\n        \"\"\"\n        Create an index from metadata.\n\n        The input is a list or dictionary that contains metadata\n        in the format identified by metadata_format_name.\n\n        The output should be a set of key-value pairs that represent\n        the information stored in `metadata´.\n\n        Parameters\n        ----------\n        metadata : Dict or List\n          Metadata created by an extractor.\n\n        Returns\n        ----------\n        dict:\n           key-value pairs representing the information in metadata.\n           values can be literals or lists of literals\n        \"\"\"\n        raise NotImplementedError",
                            "ast_type_str": "ClassDef",
                            "start_lineno": 25,
                            "end_lineno": 46,
                            "parent": {
                                "file_name": "datalad/metadata/indexers/base.py",
                                "type": "class",
                                "name": "MetadataIndexer",
                                "content": "class MetadataIndexer(object):\n    \"\"\" Defines the indexer interface \"\"\"\n    def __init__(self, metadata_format_name: str):\n        \"\"\"\n        Create a metadata indexer\n\n        The format name is passed to the constructor to allow\n        a single indexer to process multiple extractor results\n        \"\"\"\n        self.metadata_format_name = metadata_format_name\n\n    def create_index(self, metadata: Union[Dict, List]) -> Dict[str, Any]:\n        \"\"\"\n        Create an index from metadata.\n\n        The input is a list or dictionary that contains metadata\n        in the format identified by metadata_format_name.\n\n        The output should be a set of key-value pairs that represent\n        the information stored in `metadata´.\n\n        Parameters\n        ----------\n        metadata : Dict or List\n          Metadata created by an extractor.\n\n        Returns\n        ----------\n        dict:\n           key-value pairs representing the information in metadata.\n           values can be literals or lists of literals\n        \"\"\"\n        raise NotImplementedError",
                                "ast_type_str": "ClassDef",
                                "start_lineno": 14,
                                "end_lineno": 46,
                                "parent": null
                            }
                        },
                        {
                            "file_name": "datalad/metadata/indexers/base.py",
                            "type": "class",
                            "name": "MetadataIndexer",
                            "content": "class MetadataIndexer(object):\n    \"\"\" Defines the indexer interface \"\"\"\n    def __init__(self, metadata_format_name: str):\n        \"\"\"\n        Create a metadata indexer\n\n        The format name is passed to the constructor to allow\n        a single indexer to process multiple extractor results\n        \"\"\"\n        self.metadata_format_name = metadata_format_name\n\n    def create_index(self, metadata: Union[Dict, List]) -> Dict[str, Any]:\n        \"\"\"\n        Create an index from metadata.\n\n        The input is a list or dictionary that contains metadata\n        in the format identified by metadata_format_name.\n\n        The output should be a set of key-value pairs that represent\n        the information stored in `metadata´.\n\n        Parameters\n        ----------\n        metadata : Dict or List\n          Metadata created by an extractor.\n\n        Returns\n        ----------\n        dict:\n           key-value pairs representing the information in metadata.\n           values can be literals or lists of literals\n        \"\"\"\n        raise NotImplementedError",
                            "ast_type_str": "ClassDef",
                            "start_lineno": 14,
                            "end_lineno": 46,
                            "parent": null
                        }
                    ],
                    "added_entities": [],
                    "deleted_entities": []
                }
            ]
        },
        {
            "old_file_content": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Interface for managing metadata\n\"\"\"\n\n__docformat__ = 'restructuredtext'\n\nimport collections\nimport logging\nfrom datalad.log import log_progress\nlgr = logging.getLogger('datalad.metadata.search')\n\nimport os\nimport re\nfrom functools import partial\nfrom os.path import join as opj, exists\nfrom os.path import relpath\nfrom os.path import normpath\nimport sys\nfrom time import time\n\nfrom pkg_resources import EntryPoint, iter_entry_points\n\nfrom datalad import cfg\nfrom datalad.interface.base import Interface\nfrom datalad.interface.base import build_doc\nfrom datalad.interface.utils import eval_results\nfrom datalad.distribution.dataset import Dataset\nfrom datalad.distribution.dataset import datasetmethod, EnsureDataset, \\\n    require_dataset\nfrom datalad.support.gitrepo import GitRepo\nfrom datalad.support.param import Parameter\nfrom datalad.support.constraints import EnsureNone\nfrom datalad.support.constraints import EnsureInt\n\nfrom datalad.consts import SEARCH_INDEX_DOTGITDIR\nfrom datalad.utils import (\n    as_unicode,\n    ensure_list,\n    ensure_unicode,\n    get_suggestions_msg,\n    shortened_repr,\n    unicode_srctypes,\n)\nfrom datalad.support.exceptions import NoDatasetFound\nfrom datalad.ui import ui\nfrom datalad.dochelpers import single_or_plural\nfrom datalad.dochelpers import exc_str\nfrom datalad.metadata.metadata import query_aggregated_metadata\n\n# TODO: consider using plain as_unicode, without restricting\n# the types?\n_any2unicode = partial(as_unicode, cast_types=(int, float, tuple, list, dict))\n\n\ndef _listdict2dictlist(lst, strict=True):\n    \"\"\"Helper to deal with DataLad's unique value reports\n\n    Parameters\n    ----------\n    lst : list\n      List of dicts\n    strict : bool\n      In strict mode any dictionary items that doesn't have a simple value\n      (but another container) is discarded. In non-strict mode, arbitrary\n      levels of nesting are preserved, and no unique-ification of values\n      is performed. The latter can be used when it is mostly (or merely)\n      interesting to get a list of metadata keys.\n    \"\"\"\n    # unique values that we got, always a list\n    if all(not isinstance(uval, dict) for uval in lst):\n        # no nested structures, take as is\n        return lst\n\n    type_excluder = (tuple, list)\n    if strict:\n        type_excluder += (dict,)\n    # we need to turn them inside out, instead of a list of\n    # dicts, we want a dict where values are lists, because we\n    # cannot handle hierarchies in a tabular search index\n    # if you came here to find that out, go ahead and use a graph\n    # DB/search\n    udict = {}\n    for uv in lst:\n        if not isinstance(uv, dict):\n            # we cannot mix and match, discard any scalar values\n            # in favor of the structured metadata\n            continue\n        for k, v in uv.items():\n            if isinstance(v, type_excluder):\n                # this is where we draw the line, two levels of\n                # nesting. whoosh can only handle string values\n                # injecting a stringified blob of something doesn't\n                # really enable anything useful -> graph search\n                continue\n            if v == \"\":\n                # no cruft\n                continue\n            if strict:\n                # no duplicate values, only hashable stuff\n                uvals = udict.get(k, set())\n                uvals.add(v)\n            else:\n                # whatever it is, we'll take it\n                uvals = udict.get(k, [])\n                uvals.append(v)\n            udict[k] = uvals\n    return {\n        # set not good for JSON, have plain list\n        k: list(v) if len(v) > 1 else list(v)[0]\n        for k, v in udict.items()\n        # do not accumulate cruft\n        if len(v)}\n\n\ndef _meta2autofield_dict(meta, val2str=True, schema=None, consider_ucn=True):\n    \"\"\"Takes care of dtype conversion into unicode, potential key mappings\n    and concatenation of sequence-type fields into CSV strings\n\n    - if `consider_ucn` (default) it would copy keys from\n      datalad_unique_content_properties into `meta` for that extractor\n    - ... TODO ...\n    \"\"\"\n    if consider_ucn:\n        # loop over all metadata sources and the report of their unique values\n        ucnprops = meta.get(\"datalad_unique_content_properties\", {})\n        for src, umeta in ucnprops.items():\n            srcmeta = meta.get(src, {})\n            for uk in umeta:\n                if uk in srcmeta:\n                    # we have a real entry for this key in the dataset metadata\n                    # ignore any generated unique value list in favor of the\n                    # tailored data\n                    continue\n                srcmeta[uk] = _listdict2dictlist(umeta[uk], strict=False) if umeta[uk] is not None else None\n            if src not in meta and srcmeta:\n                meta[src] = srcmeta  # assign the new one back\n\n    srcmeta = None   # for paranoids to avoid some kind of manipulation of the last\n\n    def _deep_kv(basekey, dct):\n        \"\"\"Return key/value pairs of any depth following a rule for key\n        composition\n\n        dct must be a dict\n        \"\"\"\n        for k, v in dct.items():\n            if (k != '@id' and k.startswith('@')) or k == 'datalad_unique_content_properties':\n                # ignore all JSON-LD specials, but @id\n                continue\n            # TODO `k` might need remapping, if another key was already found\n            # with the same definition\n            key = u'{}{}'.format(\n                basekey,\n                # replace now special chars, and avoid spaces\n                # `os.sep` needs to go, because whoosh uses the field name for\n                # temp files during index staging, and trips over absent \"directories\"\n                # TODO maybe even kill parentheses\n                # TODO actually, it might be better to have an explicit whitelist\n                k.lstrip('@').replace(os.sep, '_').replace(' ', '_').replace('-', '_').replace('.', '_').replace(':', '-')\n            )\n            if isinstance(v, list):\n                v = _listdict2dictlist(v)\n\n            if isinstance(v, dict):\n                # dive\n                for i in _deep_kv('{}.'.format(key), v):\n                    yield i\n            else:\n                yield key, v\n\n    def get_indexer_for(metadata_format_name: str) -> callable:\n        indexer = (tuple(iter_entry_points('datalad.metadata.indexers', metadata_format_name)) or (None,))[0]\n        if isinstance(indexer, EntryPoint):\n            try:\n                indexer_object = indexer.load()(metadata_format_name)\n                return indexer_object.create_index\n            except Exception as e:\n                lgr.warning('Failed to load indexer %s: %s', indexer.name, exc_str(e))\n        return lambda metadata: _deep_kv('', metadata)\n\n    def _iterable_to_unicode_string(value, perform_operation):\n        if perform_operation:\n            if isinstance(value, (list, tuple)):\n                return u' '.join(_any2unicode(i) for i in value)\n            return _any2unicode(value)\n        return value\n\n    meta = meta or {}\n    return {\n        # Collect all meta-items which have a non-dict value type and where\n        # the key not absent in a given schema.\n        **{\n            key: _iterable_to_unicode_string(value, val2str)\n            for key, value in filter(lambda kv: not isinstance(kv[1], dict), meta.items())\n            if schema is None or key in schema\n        },\n\n        # Collect all meta-items which have a dict value type and where\n        # the key is neither 'datalad_unique_content_properties' nor absent\n        # in a given schema.\n        # These values are considered as metadata, the keys are considered to\n        # be the name of the extractor, i.e. the metadata_format_name, that created\n        # the metadata.\n        **{\n            metadata_format_name + '.' + sub_key: _iterable_to_unicode_string(sub_key_value, val2str)\n            for metadata_format_name, metadata_content in filter(\n                lambda kv: isinstance(kv[1], dict) and kv[0] != 'datalad_unique_content_properties',\n                meta.items()\n            )\n            for sub_key, sub_key_value in get_indexer_for(metadata_format_name)(metadata_content)\n            if schema is None or metadata_format_name + '.' + sub_key in schema\n        }\n    }\n\n\ndef _search_from_virgin_install(dataset, query):\n    #\n    # this is to be nice to newbies\n    #\n    exc_info = sys.exc_info()\n    if dataset is None:\n        if not ui.is_interactive:\n            raise NoDatasetFound(\n                \"No DataLad dataset found. Specify a dataset to be \"\n                \"searched, or run interactively to get assistance \"\n                \"installing a queriable superdataset.\"\n            )\n        # none was provided so we could ask user whether he possibly wants\n        # to install our beautiful mega-duper-super-dataset?\n        # TODO: following logic could possibly benefit other actions.\n        DEFAULT_DATASET_PATH = cfg.obtain('datalad.locations.default-dataset')\n        if os.path.exists(DEFAULT_DATASET_PATH):\n            default_ds = Dataset(DEFAULT_DATASET_PATH)\n            if default_ds.is_installed():\n                if ui.yesno(\n                    title=\"No DataLad dataset found at current location\",\n                    text=\"Would you like to search the DataLad \"\n                         \"superdataset at %r?\"\n                          % DEFAULT_DATASET_PATH):\n                    pass\n                else:\n                    raise exc_info[1]\n            else:\n                raise NoDatasetFound(\n                    \"No DataLad dataset found at current location. \"\n                    \"The DataLad superdataset location %r exists, \"\n                    \"but does not contain an dataset.\"\n                    % DEFAULT_DATASET_PATH)\n        elif ui.yesno(\n                title=\"No DataLad dataset found at current location\",\n                text=\"Would you like to install the DataLad \"\n                     \"superdataset at %r?\"\n                     % DEFAULT_DATASET_PATH):\n            from datalad.api import install\n            default_ds = install(DEFAULT_DATASET_PATH, source='///')\n            ui.message(\n                \"From now on you can refer to this dataset using the \"\n                \"label '///'\"\n            )\n        else:\n            raise exc_info[1]\n\n        lgr.info(\n            \"Performing search using DataLad superdataset %r\",\n            default_ds.path\n        )\n        for res in default_ds.search(query):\n            yield res\n        return\n    else:\n        raise  # this function is called within exception handling block\n\n\nclass _Search(object):\n    def __init__(self, ds, **kwargs):\n        self.ds = ds\n        self.documenttype = self.ds.config.obtain(\n            'datalad.search.index-{}-documenttype'.format(self._mode_label),\n            default=self._default_documenttype)\n\n    def __call__(self, query, max_nresults=None):\n        raise NotImplementedError\n\n    @classmethod\n    def _key_matches(cls, k, regexes):\n        \"\"\"Return which regex the key matches\n        \"\"\"\n        for regex in regexes:\n            if re.search(regex, k):\n                return regex\n\n    def show_keys(self, *args, **kwargs):\n        raise NotImplementedError(args)\n\n    def get_query(self, query):\n        \"\"\"Prepare query structure specific for a search backend.\n\n        It can also memorize within instance some parameters of the last query\n        which could be used to assist output formatting/structuring later on\n        \"\"\"\n        raise NotImplementedError\n\n    def get_nohits_msg(self):\n        \"\"\"Given what it knows, provide recommendation in the case of no hits\"\"\"\n        return \"No search hits, wrong query? \" \\\n               \"See 'datalad search --show-keys name' for known keys \" \\\n               \"and 'datalad search --help' on how to prepare your query.\"\n\n\nclass _WhooshSearch(_Search):\n    def __init__(self, ds, force_reindex=False, **kwargs):\n        super(_WhooshSearch, self).__init__(ds, **kwargs)\n\n        self.idx_obj = None\n        # where does the bunny have the eggs?\n\n        self.index_dir = opj(str(self.ds.repo.dot_git), SEARCH_INDEX_DOTGITDIR)\n        self._mk_search_index(force_reindex)\n\n    def show_keys(self, mode, regexes=None):\n        \"\"\"\n\n        Parameters\n        ----------\n        mode: {\"name\"}\n        regexes: list of regex\n          Which keys to bother working on\n        \"\"\"\n        if mode != 'name':\n            raise NotImplementedError(\n                \"ATM %s can show only names, so please use show_keys with 'name'\"\n                % self.__class__.__name__\n            )\n        for k in self.idx_obj.schema.names():\n            if regexes and not self._key_matches(k, regexes):\n                continue\n            print(u'{}'.format(k))\n\n    def get_query(self, query):\n        # parse the query string\n        self._mk_parser()\n        # for convenience we accept any number of args-words from the\n        # shell and put them together to a single string here\n        querystr = ' '.join(ensure_list(query))\n        # this gives a formal whoosh query\n        wquery = self.parser.parse(querystr)\n        return wquery\n\n    def _meta2doc(self, meta, val2str=True, schema=None):\n        raise NotImplementedError\n\n    def _mk_schema(self):\n        raise NotImplementedError\n\n    def _mk_parser(self):\n        raise NotImplementedError\n\n    def _mk_search_index(self, force_reindex):\n        \"\"\"Generic entrypoint to index generation\n\n        The actual work that determines the structure and content of the index\n        is done by functions that are passed in as arguments\n\n        `meta2doc` - must return dict for index document from result input\n        \"\"\"\n        from whoosh import index as widx\n        from .metadata import get_ds_aggregate_db_locations\n        dbloc, db_base_path = get_ds_aggregate_db_locations(self.ds)\n        # what is the lastest state of aggregated metadata\n        metadata_state = self.ds.repo.get_last_commit_hexsha(relpath(dbloc, start=self.ds.path))\n        # use location common to all index types, they would all invalidate\n        # simultaneously\n        stamp_fname = opj(self.index_dir, 'datalad_metadata_state')\n        index_dir = opj(self.index_dir, self._mode_label)\n\n        if (not force_reindex) and \\\n                exists(index_dir) and \\\n                exists(stamp_fname) and \\\n                open(stamp_fname).read() == metadata_state:\n            try:\n                # TODO check that the index schema is the same\n                # as the one we would have used for reindexing\n                # TODO support incremental re-indexing, whoosh can do it\n                idx = widx.open_dir(index_dir)\n                lgr.debug(\n                    'Search index contains %i documents',\n                    idx.doc_count())\n                self.idx_obj = idx\n                return\n            except widx.LockError as e:\n                raise e\n            except widx.IndexError as e:\n                # Generic index error.\n                # we try to regenerate\n                lgr.warning(\n                    \"Cannot open existing index %s (%s), will regenerate\",\n                    index_dir, exc_str(e)\n                )\n            except widx.IndexVersionError as e:  # (msg, version, release=None)\n                # Raised when you try to open an index using a format that the\n                # current version of Whoosh cannot read. That is, when the index\n                # you're trying to open is either not backward or forward\n                # compatible with this version of Whoosh.\n                # we try to regenerate\n                lgr.warning(exc_str(e))\n                pass\n            except widx.OutOfDateError as e:\n                # Raised when you try to commit changes to an index which is not\n                # the latest generation.\n                # this should not happen here, but if it does ... KABOOM\n                raise\n            except widx.EmptyIndexError as e:\n                # Raised when you try to work with an index that has no indexed\n                # terms.\n                # we can just continue with generating an index\n                pass\n            except ValueError as e:\n                if 'unsupported pickle protocol' in str(e):\n                    lgr.warning(\n                        \"Cannot open existing index %s (%s), will regenerate\",\n                        index_dir, exc_str(e)\n                    )\n                else:\n                    raise\n\n        lgr.info('{} search index'.format(\n            'Rebuilding' if exists(index_dir) else 'Building'))\n\n        if not exists(index_dir):\n            os.makedirs(index_dir)\n\n        # this is a pretty cheap call that just pull this info from a file\n        dsinfo = self.ds.metadata(\n            get_aggregates=True,\n            return_type='list',\n            result_renderer='disabled')\n\n        self._mk_schema(dsinfo)\n\n        idx_obj = widx.create_in(index_dir, self.schema)\n        idx = idx_obj.writer(\n            # cache size per process\n            limitmb=cfg.obtain('datalad.search.indexercachesize'),\n            # disable parallel indexing for now till #1927 is resolved\n            ## number of processes for indexing\n            #procs=multiprocessing.cpu_count(),\n            ## write separate index segments in each process for speed\n            ## asks for writer.commit(optimize=True)\n            #multisegment=True,\n        )\n\n        # load metadata of the base dataset and what it knows about all its subdatasets\n        # (recursively)\n        old_idx_size = 0\n        old_ds_rpath = ''\n        idx_size = 0\n        log_progress(\n            lgr.info,\n            'autofieldidxbuild',\n            'Start building search index',\n            total=len(dsinfo),\n            label='Building search index',\n            unit=' Datasets',\n        )\n        for res in query_aggregated_metadata(\n                reporton=self.documenttype,\n                ds=self.ds,\n                aps=[dict(path=self.ds.path, type='dataset')],\n                # MIH: I cannot see a case when we would not want recursion (within\n                # the metadata)\n                recursive=True):\n            # this assumes that files are reported after each dataset report,\n            # and after a subsequent dataset report no files for the previous\n            # dataset will be reported again\n            meta = res.get('metadata', {})\n            doc = self._meta2doc(meta)\n            admin = {\n                'type': res['type'],\n                'path': relpath(res['path'], start=self.ds.path),\n            }\n            if 'parentds' in res:\n                admin['parentds'] = relpath(res['parentds'], start=self.ds.path)\n            if admin['type'] == 'dataset':\n                if old_ds_rpath:\n                    lgr.debug(\n                        'Added %s on dataset %s',\n                        single_or_plural(\n                            'document',\n                            'documents',\n                            idx_size - old_idx_size,\n                            include_count=True),\n                        old_ds_rpath)\n                log_progress(lgr.info, 'autofieldidxbuild',\n                             'Indexed dataset at %s', old_ds_rpath,\n                             update=1, increment=True)\n                old_idx_size = idx_size\n                old_ds_rpath = admin['path']\n                admin['id'] = res.get('dsid', None)\n\n            doc.update({k: ensure_unicode(v) for k, v in admin.items()})\n            lgr.debug(\"Adding document to search index: {}\".format(doc))\n            # inject into index\n            idx.add_document(**doc)\n            idx_size += 1\n\n        if old_ds_rpath:\n            lgr.debug(\n                'Added %s on dataset %s',\n                single_or_plural(\n                    'document',\n                    'documents',\n                    idx_size - old_idx_size,\n                    include_count=True),\n                old_ds_rpath)\n\n        lgr.debug(\"Committing index\")\n        idx.commit(optimize=True)\n        log_progress(\n            lgr.info, 'autofieldidxbuild', 'Done building search index')\n\n        # \"timestamp\" the search index to allow for automatic invalidation\n        with open(stamp_fname, 'w') as f:\n            f.write(metadata_state)\n\n        lgr.info('Search index contains %i documents', idx_size)\n        self.idx_obj = idx_obj\n\n    def __call__(self, query, max_nresults=None, force_reindex=False, full_record=False):\n        if max_nresults is None:\n            # mode default\n            max_nresults = 20\n\n        with self.idx_obj.searcher() as searcher:\n            wquery = self.get_query(query)\n\n            # perform the actual search\n            hits = searcher.search(\n                wquery,\n                terms=True,\n                limit=max_nresults if max_nresults > 0 else None)\n            # report query stats\n            topstr = '{} top {}'.format(\n                max_nresults,\n                single_or_plural('match', 'matches', max_nresults)\n            )\n            lgr.info('Query completed in {} sec.{}'.format(\n                hits.runtime,\n                ' Reporting {}.'.format(\n                    ('up to ' + topstr)\n                    if max_nresults > 0\n                    else 'all matches'\n                )\n                if not hits.is_empty()\n                else ' No matches.'\n            ))\n\n            if not hits:\n                return\n\n            nhits = 0\n            annotated_hits = []\n            # annotate hits for full metadata report\n            for i, hit in enumerate(hits):\n                annotated_hit = dict(\n                    path=normpath(opj(self.ds.path, hit['path'])),\n                    query_matched={ensure_unicode(k): ensure_unicode(v)\n                                   if isinstance(v, unicode_srctypes) else v\n                                   for k, v in hit.matched_terms()},\n                    parentds=normpath(\n                        opj(self.ds.path, hit['parentds'])) if 'parentds' in hit else None,\n                    type=hit.get('type', None))\n                if not full_record:\n                    annotated_hit.update(\n                        refds=self.ds.path,\n                        action='search',\n                        status='ok',\n                        logger=lgr,\n                    )\n                    yield annotated_hit\n                else:\n                    annotated_hits.append(annotated_hit)\n                nhits += 1\n            if full_record:\n                for res in query_aggregated_metadata(\n                        # type is taken from hit record\n                        reporton=None,\n                        ds=self.ds,\n                        aps=annotated_hits,\n                        # never recursive, we have direct hits already\n                        recursive=False):\n                    res.update(\n                        refds=self.ds.path,\n                        action='search',\n                        status='ok',\n                        logger=lgr,\n                    )\n                    yield res\n\n            if max_nresults and nhits == max_nresults:\n                lgr.info(\n                    \"Reached the limit of {}, there could be more which \"\n                    \"were not reported.\".format(topstr)\n                )\n\n\nclass _BlobSearch(_WhooshSearch):\n    _mode_label = 'textblob'\n    _default_documenttype = 'datasets'\n\n    def _meta2doc(self, meta):\n        # coerce the entire flattened metadata dict into a comma-separated string\n        # that also includes the keys\n        return dict(meta=u', '.join(\n            u'{}: {}'.format(k, v)\n            for k, v in _meta2autofield_dict(\n                meta,\n                val2str=True,\n                schema=None).items()))\n\n    def _mk_schema(self, dsinfo):\n        from whoosh import fields as wf\n        from whoosh.analysis import StandardAnalyzer\n\n        # TODO support some customizable mapping to homogenize some metadata fields\n        # onto a given set of index keys\n        self.schema = wf.Schema(\n            id=wf.ID,\n            path=wf.ID(stored=True),\n            type=wf.ID(stored=True),\n            parentds=wf.ID(stored=True),\n            meta=wf.TEXT(\n                stored=False,\n                analyzer=StandardAnalyzer(minsize=2))\n        )\n\n    def _mk_parser(self):\n        from whoosh import qparser as qparse\n\n        # use whoosh default query parser for now\n        parser = qparse.QueryParser(\n            \"meta\",\n            schema=self.idx_obj.schema\n        )\n        parser.add_plugin(qparse.FuzzyTermPlugin())\n        parser.remove_plugin_class(qparse.PhrasePlugin)\n        parser.add_plugin(qparse.SequencePlugin())\n        self.parser = parser\n\n\nclass _AutofieldSearch(_WhooshSearch):\n    _mode_label = 'autofield'\n    _default_documenttype = 'all'\n\n    def _meta2doc(self, meta):\n        return _meta2autofield_dict(meta, val2str=True, schema=self.schema)\n\n    def _mk_schema(self, dsinfo):\n        from whoosh import fields as wf\n        from whoosh.analysis import SimpleAnalyzer\n\n        # haven for terms that have been found to be undefined\n        # (for faster decision-making upon next encounter)\n        # this will harvest all discovered term definitions\n        definitions = {\n            '@id': 'unique identifier of an entity',\n            # TODO make proper JSON-LD definition\n            'path': 'path name of an entity relative to the searched base dataset',\n            # TODO make proper JSON-LD definition\n            'parentds': 'path of the datasets that contains an entity',\n            # 'type' will not come from a metadata field, hence will not be detected\n            'type': 'type of a record',\n        }\n\n        schema_fields = {\n            n.lstrip('@'): wf.ID(stored=True, unique=n == '@id')\n            for n in definitions}\n\n        lgr.debug('Scanning for metadata keys')\n        # quick 1st pass over all dataset to gather the needed schema fields\n        log_progress(\n            lgr.info,\n            'idxschemabuild',\n            'Start building search schema',\n            total=len(dsinfo),\n            label='Building search schema',\n            unit=' Datasets',\n        )\n        for res in query_aggregated_metadata(\n                # XXX TODO After #2156 datasets may not necessarily carry all\n                # keys in the \"unique\" summary\n                reporton='datasets',\n                ds=self.ds,\n                aps=[dict(path=self.ds.path, type='dataset')],\n                recursive=True):\n            meta = res.get('metadata', {})\n            # no stringification of values for speed, we do not need/use the\n            # actual values at this point, only the keys\n            idxd = _meta2autofield_dict(meta, val2str=False)\n\n            for k in idxd:\n                schema_fields[k] = wf.TEXT(stored=False,\n                                           analyzer=SimpleAnalyzer())\n            log_progress(lgr.info, 'idxschemabuild',\n                         'Scanned dataset at %s', res['path'],\n                         update=1, increment=True)\n        log_progress(\n            lgr.info, 'idxschemabuild', 'Done building search schema')\n\n        self.schema = wf.Schema(**schema_fields)\n\n    def _mk_parser(self):\n        from whoosh import qparser as qparse\n\n        parser = qparse.MultifieldParser(\n            self.idx_obj.schema.names(),\n            self.idx_obj.schema)\n        # XXX: plugin is broken in Debian's whoosh 2.7.0-2, but already fixed\n        # upstream\n        parser.add_plugin(qparse.FuzzyTermPlugin())\n        parser.add_plugin(qparse.GtLtPlugin())\n        parser.add_plugin(qparse.SingleQuotePlugin())\n        # replace field defintion to allow for colons to be part of a field's name:\n        parser.replace_plugin(qparse.FieldsPlugin(expr=r\"(?P<text>[()<>.\\w]+|[*]):\"))\n        self.parser = parser\n\n\nclass _EGrepCSSearch(_Search):\n    _mode_label = 'egrepcs'\n    _default_documenttype = 'datasets'\n\n    def __init__(self, ds, **kwargs):\n        super(_EGrepCSSearch, self).__init__(ds, **kwargs)\n        self._queried_keys = None  # to be memoized by get_query\n\n    # If there were custom \"per-search engine\" options, we could expose\n    # --consider_ucn - search through unique content properties of the dataset\n    #    which might be more computationally demanding\n    def __call__(self, query, max_nresults=None, consider_ucn=False, full_record=True):\n        if max_nresults is None:\n            # no limit by default\n            max_nresults = 0\n        query = self.get_query(query)\n\n        nhits = 0\n        for res in query_aggregated_metadata(\n                reporton=self.documenttype,\n                ds=self.ds,\n                aps=[dict(path=self.ds.path, type='dataset')],\n                # MIH: I cannot see a case when we would not want recursion (within\n                # the metadata)\n                recursive=True):\n            # this assumes that files are reported after each dataset report,\n            # and after a subsequent dataset report no files for the previous\n            # dataset will be reported again\n            meta = res.get('metadata', {})\n            # produce a flattened metadata dict to search through\n            doc = _meta2autofield_dict(meta, val2str=True, consider_ucn=consider_ucn)\n            # inject a few basic properties into the dict\n            # analog to what the other modes do in their index\n            doc.update({\n                k: res[k] for k in ('@id', 'type', 'path', 'parentds')\n                if k in res})\n            # use search instead of match to not just get hits at the start of the string\n            # this will be slower, but avoids having to use actual regex syntax at the user\n            # side even for simple queries\n            # DOTALL is needed to handle multiline description fields and such, and still\n            # be able to match content coming for a later field\n            lgr.log(7, \"Querying %s among %d items\", query, len(doc))\n            t0 = time()\n            matches = {(q['query'] if isinstance(q, dict) else q, k):\n                       q['query'].search(v) if isinstance(q, dict) else q.search(v)\n                       for k, v in doc.items()\n                       for q in query\n                       if not isinstance(q, dict) or q['field'].match(k)}\n            dt = time() - t0\n            lgr.log(7, \"Finished querying in %f sec\", dt)\n            # retain what actually matched\n            matched = {k[1]: match.group() for k, match in matches.items() if match}\n            # implement AND behavior across query expressions, but OR behavior\n            # across queries matching multiple fields for a single query expression\n            # for multiple queries, this makes it consistent with a query that\n            # has no field specification\n            if matched and len(query) == len(set(k[0] for k in matches if matches[k])):\n                hit = dict(\n                    res,\n                    action='search',\n                    query_matched=matched,\n                )\n                yield hit\n                nhits += 1\n                if max_nresults and nhits == max_nresults:\n                    # report query stats\n                    topstr = '{} top {}'.format(\n                        max_nresults,\n                        single_or_plural('match', 'matches', max_nresults)\n                    )\n                    lgr.info(\n                        \"Reached the limit of {}, there could be more which \"\n                        \"were not reported.\".format(topstr)\n                    )\n                    break\n\n    def show_keys(self, mode=None, regexes=None):\n        \"\"\"\n\n        Parameters\n        ----------\n        mode: {\"name\", \"short\", \"full\"}\n        regexes: list of regex\n          Which keys to bother working on\n        \"\"\"\n        maxl = 100  # approx maximal line length for unique values in mode=short\n        # use a dict already, later we need to map to a definition\n        # meanwhile map to the values\n\n        keys = self._get_keys(mode)\n\n        for k in sorted(keys):\n            if regexes and not self._key_matches(k, regexes):\n                continue\n            if mode == 'name':\n                print(k)\n                continue\n\n            # do a bit more\n            stat = keys[k]\n            all_uvals = uvals = sorted(stat.uvals)\n\n            stat.uvals_str = ensure_unicode(\n                \"{} unique values: \".format(len(all_uvals))\n            )\n\n            if mode == 'short':\n                # show only up until we fill maxl\n                uvals_str = ''\n                uvals = []\n                for v in all_uvals:\n                    appendix = ('; ' if uvals else '') + v\n                    if len(uvals_str) + len(appendix) > maxl - len(stat.uvals_str):\n                        break\n                    uvals.append(v)\n                    uvals_str += appendix\n            elif mode == 'full':\n                pass\n            else:\n                raise ValueError(\n                    \"Unknown value for stats. Know full and short\")\n\n            stat.uvals_str += '; '.join(uvals)\n\n            if len(all_uvals) > len(uvals):\n                stat.uvals_str += \\\n                    '; +%s' % single_or_plural(\"value\", \"values\", len(all_uvals) - len(uvals), True)\n\n            print(\n                u'{k}\\n in  {stat.ndatasets} datasets\\n has {stat.uvals_str}'.format(\n                k=k, stat=stat\n            ))\n        # After #2156 datasets may not necessarily carry all\n        # keys in the \"unique\" summary\n        lgr.warning('In this search mode, the reported list of metadata keys may be incomplete')\n\n\n    def _get_keys(self, mode=None):\n        \"\"\"Return keys and their statistics if mode != 'name'.\"\"\"\n        class key_stat:\n            def __init__(self):\n                self.ndatasets = 0  # how many datasets have this field\n                self.uvals = set()\n\n        from collections import defaultdict\n        keys = defaultdict(key_stat)\n        for res in query_aggregated_metadata(\n                # XXX TODO After #2156 datasets may not necessarily carry all\n                # keys in the \"unique\" summary\n                reporton='datasets',\n                ds=self.ds,\n                aps=[dict(path=self.ds.path, type='dataset')],\n                recursive=True):\n            meta = res.get('metadata', {})\n            # inject a few basic properties into the dict\n            # analog to what the other modes do in their index\n            meta.update({\n                k: res.get(k, None) for k in ('@id', 'type', 'path', 'parentds')\n                # parentds is tricky all files will have it, but the dataset\n                # queried above might not (single dataset), let's force it in\n                if k == 'parentds' or k in res})\n\n            # no stringification of values for speed\n            idxd = _meta2autofield_dict(meta, val2str=False)\n\n            for k, kvals in idxd.items():\n                # TODO deal with conflicting definitions when available\n                keys[k].ndatasets += 1\n                if mode == 'name':\n                    continue\n                keys[k].uvals |= self.get_repr_uvalues(kvals)\n        return keys\n\n    def get_repr_uvalues(self, kvals):\n        kvals_set = set()\n        if not kvals:\n            return kvals_set\n        kvals_iter = (\n            kvals\n            if hasattr(kvals, '__iter__') and not isinstance(kvals, (str, bytes))\n            else [kvals]\n        )\n        return set(shortened_repr(x, 50) for x in kvals_iter)\n\n    def get_query(self, query):\n        query = ensure_list(query)\n        simple_fieldspec = re.compile(r\"(?P<field>\\S*?):(?P<query>.*)\")\n        quoted_fieldspec = re.compile(r\"'(?P<field>[^']+?)':(?P<query>.*)\")\n        query_rec_matches = [\n            simple_fieldspec.match(q) or\n            quoted_fieldspec.match(q) or\n            q\n            for q in query]\n        query_group_dicts_only = [\n            q.groupdict() for q in query_rec_matches if hasattr(q, 'groupdict')\n        ]\n        self._queried_keys = [\n            qgd['field']\n            for qgd in query_group_dicts_only\n            if ('field' in qgd and qgd['field'])\n        ]\n        if len(query_group_dicts_only) != len(query_rec_matches):\n            # we had a query element without field specification add\n            # None as an indicator of that\n            self._queried_keys.append(None)\n        # expand matches, compile expressions\n        query = [\n            {k: self._compile_query(v) for k, v in q.groupdict().items()}\n            if hasattr(q, 'groupdict') else self._compile_query(q)\n            for q in query_rec_matches\n        ]\n\n        # turn \"empty\" field specs into simple queries\n        # this is used to forcibly disable field-based search\n        # e.g. when searching for a value\n        query = [q['query']\n                 if isinstance(q, dict) and q['field'].pattern == '' else q\n                 for q in query]\n        return query\n\n    def _xfm_query(self, q):\n        # implement potential transformations of regex before they get compiled\n        return q\n\n    def _compile_query(self, q):\n        \"\"\"xfm and compile the query, with informative exception if query is incorrect\n        \"\"\"\n        q_xfmed = self._xfm_query(q)\n        try:\n            return re.compile(q_xfmed)\n        except re.error as exc:\n            omsg = \" (original: '%s')\" % q if q != q_xfmed else ''\n            raise ValueError(\n                \"regular expression '%s'%s is incorrect: %s\"\n                % (q_xfmed, omsg, exc)\n            )\n\n    def get_nohits_msg(self):\n        \"\"\"Given the query and performed search, provide recommendation\n\n        Quite often a key in the query is mistyped or I simply query for something\n        which is not actually known.  It requires --show-keys  run first, doing\n        visual search etc to mitigate.  Here we can analyze either all queried\n        keys are actually known, and if not known -- what would be the ones available.\n\n        Returns\n        -------\n        str\n          A sentence or a paragraph to be logged/output\n        \"\"\"\n        #\n        queried_keys = self._queried_keys[:]\n        if queried_keys and None in queried_keys:\n            queried_keys.pop(queried_keys.index(None))\n        if not queried_keys:\n            return  # No keys were queried, we are of no use here\n        known_keys = self._get_keys(mode='name')\n        unknown_keys = sorted(list(set(queried_keys).difference(known_keys)))\n        if not unknown_keys:\n            return  # again we are of no help here\n        msg = super(_EGrepCSSearch, self).get_nohits_msg()\n        msg += \" Following keys were not found in available metadata: %s. \" \\\n              % \", \".join(unknown_keys)\n        suggestions_msg = get_suggestions_msg(unknown_keys, known_keys)\n        if suggestions_msg:\n            msg += ' ' + suggestions_msg\n        return msg\n\n\nclass _EGrepSearch(_EGrepCSSearch):\n    _mode_label = 'egrep'\n    _default_documenttype = 'datasets'\n\n    def _xfm_query(self, q):\n        if q == q.lower():\n            # we have no upper case symbol in the query, go case-insensitive\n            return '(?i){}'.format(q)\n        else:\n            return q\n\n\n@build_doc\nclass Search(Interface):\n    \"\"\"Search dataset metadata\n\n    DataLad can search metadata extracted from a dataset and/or aggregated into\n    a superdataset (see the `aggregate-metadata` command). This makes it\n    possible to discover datasets, or individual files in a dataset even when\n    they are not available locally.\n\n    Ultimately DataLad metadata are a graph of linked data structures. However,\n    this command does not (yet) support queries that can exploit all\n    information stored in the metadata. At the moment the following search\n    modes are implemented that represent different trade-offs between the\n    expressiveness of a query and the computational and storage resources\n    required to execute a query.\n\n    - egrep (default)\n\n    - egrepcs [case-sensitive egrep]\n\n    - textblob\n\n    - autofield\n\n    An alternative default mode can be configured by tuning the\n    configuration variable 'datalad.search.default-mode'::\n\n      [datalad \"search\"]\n        default-mode = egrepcs\n\n    Each search mode has its own default configuration for what kind of\n    documents to query. The respective default can be changed via configuration\n    variables::\n\n      [datalad \"search\"]\n        index-<mode_name>-documenttype = (all|datasets|files)\n\n\n    *Mode: egrep/egrepcs*\n\n    These search modes are largely ignorant of the metadata structure, and\n    simply perform matching of a search pattern against a flat\n    string-representation of metadata. This is advantageous when the query is\n    simple and the metadata structure is irrelevant, or precisely known.\n    Moreover, it does not require a search index, hence results can be reported\n    without an initial latency for building a search index when the underlying\n    metadata has changed (e.g. due to a dataset update). By default, these\n    search modes only consider datasets and do not investigate records for\n    individual files for speed reasons. Search results are reported in the\n    order in which they were discovered.\n\n    Queries can make use of Python regular expression syntax\n    (https://docs.python.org/3/library/re.html). In `egrep` mode, matching is\n    case-insensitive when the query does not contain upper case characters, but\n    is case-sensitive when it does. In `egrepcs` mode, matching is always\n    case-sensitive. Expressions will match anywhere in a metadata string, not\n    only at the start.\n\n    When multiple queries are given, all queries have to match for a search hit\n    (AND behavior).\n\n    It is possible to search individual metadata key/value items by prefixing\n    the query with a metadata key name, separated by a colon (':'). The key\n    name can also be a regular expression to match multiple keys. A query match\n    happens when any value of an item with a matching key name matches the query\n    (OR behavior). See examples for more information.\n\n    Examples:\n\n      Query for (what happens to be) an author::\n\n        % datalad search haxby\n\n      Queries are case-INsensitive when the query contains no upper case characters,\n      and can be regular expressions. Use `egrepcs` mode when it is desired\n      to perform a case-sensitive lowercase match::\n\n        % datalad search --mode egrepcs halchenko.*haxby\n\n      This search mode performs NO analysis of the metadata content.  Therefore\n      queries can easily fail to match. For example, the above query implicitly\n      assumes that authors are listed in alphabetical order.  If that is the\n      case (which may or may not be true), the following query would yield NO\n      hits::\n\n        % datalad search Haxby.*Halchenko\n\n      The ``textblob`` search mode represents an alternative that is more\n      robust in such cases.\n\n      For more complex queries multiple query expressions can be provided that\n      all have to match to be considered a hit (AND behavior). This query\n      discovers all files (non-default behavior) that match 'bids.type=T1w'\n      AND 'nifti1.qform_code=scanner'::\n\n        % datalad -c datalad.search.index-egrep-documenttype=all search bids.type:T1w nifti1.qform_code:scanner\n\n      Key name selectors can also be expressions, which can be used to select\n      multiple keys or construct \"fuzzy\" queries. In such cases a query matches\n      when any item with a matching key matches the query (OR behavior).\n      However, multiple queries are always evaluated using an AND conjunction.\n      The following query extends the example above to match any files that\n      have either 'nifti1.qform_code=scanner' or 'nifti1.sform_code=scanner'::\n\n        % datalad -c datalad.search.index-egrep-documenttype=all search bids.type:T1w nifti1.(q|s)form_code:scanner\n\n    *Mode: textblob*\n\n    This search mode is very similar to the ``egrep`` mode, but with a few key\n    differences. A search index is built from the string-representation of\n    metadata records. By default, only datasets are included in this index, hence\n    the indexing is usually completed within a few seconds, even for hundreds\n    of datasets. This mode uses its own query language (not regular expressions)\n    that is similar to other search engines. It supports logical conjunctions\n    and fuzzy search terms. More information on this is available from the Whoosh\n    project (search engine implementation):\n\n      - Description of the Whoosh query language:\n        http://whoosh.readthedocs.io/en/latest/querylang.html)\n\n      - Description of a number of query language customizations that are\n        enabled in DataLad, such as, fuzzy term matching:\n        http://whoosh.readthedocs.io/en/latest/parsing.html#common-customizations\n\n    Importantly, search hits are scored and reported in order of descending\n    relevance, hence limiting the number of search results is more meaningful\n    than in the 'egrep' mode and can also reduce the query duration.\n\n    Examples:\n\n      Search for (what happens to be) two authors, regardless of the order in\n      which those names appear in the metadata::\n\n        % datalad search --mode textblob halchenko haxby\n\n      Fuzzy search when you only have an approximate idea what you are looking\n      for or how it is spelled::\n\n        % datalad search --mode textblob haxbi~\n\n      Very fuzzy search, when you are basically only confident about the first\n      two characters and how it sounds approximately (or more precisely: allow\n      for three edits and require matching of the first two characters)::\n\n        % datalad search --mode textblob haksbi~3/2\n\n      Combine fuzzy search with logical constructs::\n\n        % datalad search --mode textblob 'haxbi~ AND (hanke OR halchenko)'\n\n\n    *Mode: autofield*\n\n    This mode is similar to the 'textblob' mode, but builds a vastly more\n    detailed search index that represents individual metadata variables as\n    individual fields. By default, this search index includes records for\n    datasets and individual fields, hence it can grow very quickly into\n    a huge structure that can easily take an hour or more to build and require\n    more than a GB of storage. However, limiting it to documents on datasets\n    (see above) retains the enhanced expressiveness of queries while\n    dramatically reducing the resource demands.\n\n    Examples:\n\n      List names of search index fields (auto-discovered from the set of\n      indexed datasets) which either have a field starting with \"age\" or\n      \"gender\"::\n\n        % datalad search --mode autofield --show-keys name '\\.age' '\\.gender'\n\n      Fuzzy search for datasets with an author that is specified in a particular\n      metadata field::\n\n        % datalad search --mode autofield bids.author:haxbi~ type:dataset\n\n      Search for individual files that carry a particular description\n      prefix in their 'nifti1' metadata::\n\n        % datalad search --mode autofield nifti1.description:FSL* type:file\n\n\n    *Reporting*\n\n    Search hits are returned as standard DataLad results. On the command line\n    the '--output-format' (or '-f') option can be used to tweak results for\n    further processing.\n\n    Examples:\n\n      Format search hits as a JSON stream (one hit per line)::\n\n        % datalad -f json search haxby\n\n      Custom formatting: which terms matched the query of particular\n      results. Useful for investigating fuzzy search results::\n\n        $ datalad -f '{path}: {query_matched}' search --mode autofield bids.author:haxbi~\n    \"\"\"\n    _params_ = dict(\n        dataset=Parameter(\n            args=(\"-d\", \"--dataset\"),\n            doc=\"\"\"specify the dataset to perform the query operation on. If\n            no dataset is given, an attempt is made to identify the dataset\n            based on the current working directory and/or the `path` given\"\"\",\n            constraints=EnsureDataset() | EnsureNone()),\n        query=Parameter(\n            args=(\"query\",),\n            metavar='QUERY',\n            nargs=\"*\",\n            doc=\"\"\"query string, supported syntax and features depends on the\n            selected search mode (see documentation)\"\"\"),\n        force_reindex=Parameter(\n            args=(\"--reindex\",),\n            dest='force_reindex',\n            action='store_true',\n            doc=\"\"\"force rebuilding the search index, even if no change in the\n            dataset's state has been detected, for example, when the index\n            documenttype configuration has changed.\"\"\"),\n        max_nresults=Parameter(\n            args=(\"--max-nresults\",),\n            doc=\"\"\"maxmimum number of search results to report. Setting this\n            to 0 will report all search matches. Depending on the mode this\n            can search substantially slower. If not specified, a\n            mode-specific default setting will be used.\"\"\",\n            constraints=EnsureInt() | EnsureNone()),\n        mode=Parameter(\n            args=(\"--mode\",),\n            choices=('egrep', 'textblob', 'autofield'),\n            doc=\"\"\"Mode of search index structure and content. See section\n            SEARCH MODES for details.\"\"\"),\n        full_record=Parameter(\n            args=(\"--full-record\", '-f'),\n            action='store_true',\n            doc=\"\"\"If set, return the full metadata record for each search hit.\n            Depending on the search mode this might require additional queries.\n            By default, only data that is available to the respective search modes\n            is returned. This always includes essential information, such as the\n            path and the type.\"\"\"),\n        show_keys=Parameter(\n            args=('--show-keys',),\n            choices=('name', 'short', 'full'),\n            default=None,\n            doc=\"\"\"if given, a list of known search keys is shown. If 'name' -\n            only the name is printed one per line. If 'short' or 'full',\n            statistics (in how many datasets, and how many unique values) are\n            printed. 'short' truncates the listing of unique values.\n            QUERY, if provided, is regular expressions any of which keys should\n            contain.\n            No other action is performed (except for reindexing), even if other\n            arguments are given. Each key is accompanied by a term definition in\n            parenthesis (TODO). In most cases a definition is given in the form\n            of a URL. If an ontology definition for a term is known, this URL\n            can resolve to a webpage that provides a comprehensive definition\n            of the term. However, for speed reasons term resolution is solely done\n            on information contained in a local dataset's metadata, and definition\n            URLs might be outdated or point to no longer existing resources.\"\"\"),\n        show_query=Parameter(\n            args=('--show-query',),\n            action='store_true',\n            doc=\"\"\"if given, the formal query that was generated from the given\n            query string is shown, but not actually executed. This is mostly useful\n            for debugging purposes.\"\"\"),\n    )\n\n    @staticmethod\n    @datasetmethod(name='search')\n    @eval_results\n    def __call__(query=None,\n                 dataset=None,\n                 force_reindex=False,\n                 max_nresults=None,\n                 mode=None,\n                 full_record=False,\n                 show_keys=None,\n                 show_query=False):\n        try:\n            ds = require_dataset(dataset, check_installed=True, purpose='dataset search')\n            if ds.id is None:\n                raise NoDatasetFound(\n                    \"This does not seem to be a dataset (no DataLad dataset ID \"\n                    \"found). 'datalad create --force %s' can initialize \"\n                    \"this repository as a DataLad dataset\" % ds.path)\n        except NoDatasetFound:\n            for r in _search_from_virgin_install(dataset, query):\n                yield r\n            return\n\n        if mode is None:\n            # let's get inspired by what the dataset/user think is\n            # default\n            mode = ds.config.obtain('datalad.search.default-mode')\n\n        if mode == 'egrep':\n            searcher = _EGrepSearch\n        elif mode == 'egrepcs':\n            searcher = _EGrepCSSearch\n        elif mode == 'textblob':\n            searcher = _BlobSearch\n        elif mode == 'autofield':\n            searcher = _AutofieldSearch\n        else:\n            raise ValueError(\n                'unknown search mode \"{}\"'.format(mode))\n\n        searcher = searcher(ds, force_reindex=force_reindex)\n\n        if show_keys:\n            searcher.show_keys(show_keys, regexes=query)\n            return\n\n        if not query:\n            return\n\n        if show_query:\n            print(repr(searcher.get_query(query)))\n            return\n\n        nhits = 0\n        for r in searcher(\n                query,\n                max_nresults=max_nresults,\n                full_record=full_record):\n            nhits += 1\n            yield r\n        if not nhits:\n            lgr.info(searcher.get_nohits_msg() or 'no hits')\n",
            "new_file_content": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Interface for managing metadata\n\"\"\"\n\n__docformat__ = 'restructuredtext'\n\nimport collections\nimport logging\nfrom datalad.log import log_progress\nlgr = logging.getLogger('datalad.metadata.search')\n\nimport os\nimport re\nfrom functools import partial\nfrom os.path import join as opj, exists\nfrom os.path import relpath\nfrom os.path import normpath\nimport sys\nfrom time import time\n\nfrom datalad import cfg\nfrom datalad.interface.base import Interface\nfrom datalad.interface.base import build_doc\nfrom datalad.interface.utils import eval_results\nfrom datalad.distribution.dataset import Dataset\nfrom datalad.distribution.dataset import datasetmethod, EnsureDataset, \\\n    require_dataset\nfrom datalad.support.gitrepo import GitRepo\nfrom datalad.support.param import Parameter\nfrom datalad.support.constraints import EnsureNone\nfrom datalad.support.constraints import EnsureInt\n\nfrom datalad.consts import SEARCH_INDEX_DOTGITDIR\nfrom datalad.utils import (\n    as_unicode,\n    ensure_list,\n    ensure_unicode,\n    get_suggestions_msg,\n    shortened_repr,\n    unicode_srctypes,\n)\nfrom datalad.support.exceptions import NoDatasetFound\nfrom datalad.ui import ui\nfrom datalad.dochelpers import single_or_plural\nfrom datalad.dochelpers import exc_str\nfrom datalad.metadata.metadata import query_aggregated_metadata\n\n# TODO: consider using plain as_unicode, without restricting\n# the types?\n_any2unicode = partial(as_unicode, cast_types=(int, float, tuple, list, dict))\n\n\ndef _listdict2dictlist(lst, strict=True):\n    \"\"\"Helper to deal with DataLad's unique value reports\n\n    Parameters\n    ----------\n    lst : list\n      List of dicts\n    strict : bool\n      In strict mode any dictionary items that doesn't have a simple value\n      (but another container) is discarded. In non-strict mode, arbitrary\n      levels of nesting are preserved, and no unique-ification of values\n      is performed. The latter can be used when it is mostly (or merely)\n      interesting to get a list of metadata keys.\n    \"\"\"\n    # unique values that we got, always a list\n    if all(not isinstance(uval, dict) for uval in lst):\n        # no nested structures, take as is\n        return lst\n\n    type_excluder = (tuple, list)\n    if strict:\n        type_excluder += (dict,)\n    # we need to turn them inside out, instead of a list of\n    # dicts, we want a dict where values are lists, because we\n    # cannot handle hierarchies in a tabular search index\n    # if you came here to find that out, go ahead and use a graph\n    # DB/search\n    udict = {}\n    for uv in lst:\n        if not isinstance(uv, dict):\n            # we cannot mix and match, discard any scalar values\n            # in favor of the structured metadata\n            continue\n        for k, v in uv.items():\n            if isinstance(v, type_excluder):\n                # this is where we draw the line, two levels of\n                # nesting. whoosh can only handle string values\n                # injecting a stringified blob of something doesn't\n                # really enable anything useful -> graph search\n                continue\n            if v == \"\":\n                # no cruft\n                continue\n            if strict:\n                # no duplicate values, only hashable stuff\n                uvals = udict.get(k, set())\n                uvals.add(v)\n            else:\n                # whatever it is, we'll take it\n                uvals = udict.get(k, [])\n                uvals.append(v)\n            udict[k] = uvals\n    return {\n        # set not good for JSON, have plain list\n        k: list(v) if len(v) > 1 else list(v)[0]\n        for k, v in udict.items()\n        # do not accumulate cruft\n        if len(v)}\n\n\ndef _meta2autofield_dict(meta, val2str=True, schema=None, consider_ucn=True):\n    \"\"\"Takes care of dtype conversion into unicode, potential key mappings\n    and concatenation of sequence-type fields into CSV strings\n\n    - if `consider_ucn` (default) it would copy keys from\n      datalad_unique_content_properties into `meta` for that extractor\n    - ... TODO ...\n    \"\"\"\n    if consider_ucn:\n        # loop over all metadata sources and the report of their unique values\n        ucnprops = meta.get(\"datalad_unique_content_properties\", {})\n        for src, umeta in ucnprops.items():\n            srcmeta = meta.get(src, {})\n            for uk in umeta:\n                if uk in srcmeta:\n                    # we have a real entry for this key in the dataset metadata\n                    # ignore any generated unique value list in favor of the\n                    # tailored data\n                    continue\n                srcmeta[uk] = _listdict2dictlist(umeta[uk], strict=False) if umeta[uk] is not None else None\n            if src not in meta and srcmeta:\n                meta[src] = srcmeta  # assign the new one back\n\n    srcmeta = None   # for paranoids to avoid some kind of manipulation of the last\n\n    def _deep_kv(basekey, dct):\n        \"\"\"Return key/value pairs of any depth following a rule for key\n        composition\n\n        dct must be a dict\n        \"\"\"\n        for k, v in dct.items():\n            if (k != '@id' and k.startswith('@')) or k == 'datalad_unique_content_properties':\n                # ignore all JSON-LD specials, but @id\n                continue\n            # TODO `k` might need remapping, if another key was already found\n            # with the same definition\n            key = u'{}{}'.format(\n                basekey,\n                # replace now special chars, and avoid spaces\n                # `os.sep` needs to go, because whoosh uses the field name for\n                # temp files during index staging, and trips over absent \"directories\"\n                # TODO maybe even kill parentheses\n                # TODO actually, it might be better to have an explicit whitelist\n                k.lstrip('@').replace(os.sep, '_').replace(' ', '_').replace('-', '_').replace('.', '_').replace(':', '-')\n            )\n            if isinstance(v, list):\n                v = _listdict2dictlist(v)\n\n            if isinstance(v, dict):\n                # dive\n                for i in _deep_kv('{}.'.format(key), v):\n                    yield i\n            else:\n                yield key, v\n\n    def get_indexer(metadata_format_name: str) -> callable:\n        from pkg_resources import EntryPoint, iter_entry_points\n\n        # Use the first returned entry point, if any, because there should be\n        # at most one indexer for ``metadata_format_name´´.\n        indexer = (tuple(iter_entry_points('datalad.metadata.indexers', metadata_format_name)) or (None,))[0]\n        if isinstance(indexer, EntryPoint):\n            try:\n                indexer_object = indexer.load()(metadata_format_name)\n                return indexer_object.create_index\n            except Exception as e:\n                lgr.warning('Failed to load indexer %s: %s', indexer.name, exc_str(e))\n        return lambda metadata: _deep_kv('', metadata)\n\n    if val2str:\n        def _val2str_helper(value):\n            if isinstance(value, (list, tuple)):\n                return u' '.join(_any2unicode(i) for i in value)\n            return _any2unicode(value)\n    else:\n        def _val2str_helper(value):\n            return value\n\n    meta = meta or {}\n    return {\n        # Collect all meta-items which have a non-dict value type and where\n        # the key not absent in a given schema.\n        **{\n            key: _val2str_helper(value)\n            for key, value in filter(lambda kv: not isinstance(kv[1], dict), meta.items())\n            if schema is None or key in schema\n        },\n\n        # Collect all meta-items which have a dict value type and where\n        # the key is neither 'datalad_unique_content_properties' nor absent\n        # in a given schema.\n        # These values are considered as metadata, the keys are considered to\n        # be the name of the extractor, i.e. the metadata_format_name, that created\n        # the metadata.\n        **{\n            metadata_format_name + '.' + sub_key: _val2str_helper(sub_key_value)\n            for metadata_format_name, metadata_content in filter(\n                lambda kv: isinstance(kv[1], dict) and kv[0] != 'datalad_unique_content_properties',\n                meta.items()\n            )\n            for sub_key, sub_key_value in get_indexer(metadata_format_name)(metadata_content)\n            if schema is None or metadata_format_name + '.' + sub_key in schema\n        }\n    }\n\n\ndef _search_from_virgin_install(dataset, query):\n    #\n    # this is to be nice to newbies\n    #\n    exc_info = sys.exc_info()\n    if dataset is None:\n        if not ui.is_interactive:\n            raise NoDatasetFound(\n                \"No DataLad dataset found. Specify a dataset to be \"\n                \"searched, or run interactively to get assistance \"\n                \"installing a queriable superdataset.\"\n            )\n        # none was provided so we could ask user whether he possibly wants\n        # to install our beautiful mega-duper-super-dataset?\n        # TODO: following logic could possibly benefit other actions.\n        DEFAULT_DATASET_PATH = cfg.obtain('datalad.locations.default-dataset')\n        if os.path.exists(DEFAULT_DATASET_PATH):\n            default_ds = Dataset(DEFAULT_DATASET_PATH)\n            if default_ds.is_installed():\n                if ui.yesno(\n                    title=\"No DataLad dataset found at current location\",\n                    text=\"Would you like to search the DataLad \"\n                         \"superdataset at %r?\"\n                          % DEFAULT_DATASET_PATH):\n                    pass\n                else:\n                    raise exc_info[1]\n            else:\n                raise NoDatasetFound(\n                    \"No DataLad dataset found at current location. \"\n                    \"The DataLad superdataset location %r exists, \"\n                    \"but does not contain an dataset.\"\n                    % DEFAULT_DATASET_PATH)\n        elif ui.yesno(\n                title=\"No DataLad dataset found at current location\",\n                text=\"Would you like to install the DataLad \"\n                     \"superdataset at %r?\"\n                     % DEFAULT_DATASET_PATH):\n            from datalad.api import install\n            default_ds = install(DEFAULT_DATASET_PATH, source='///')\n            ui.message(\n                \"From now on you can refer to this dataset using the \"\n                \"label '///'\"\n            )\n        else:\n            raise exc_info[1]\n\n        lgr.info(\n            \"Performing search using DataLad superdataset %r\",\n            default_ds.path\n        )\n        for res in default_ds.search(query):\n            yield res\n        return\n    else:\n        raise  # this function is called within exception handling block\n\n\nclass _Search(object):\n    def __init__(self, ds, **kwargs):\n        self.ds = ds\n        self.documenttype = self.ds.config.obtain(\n            'datalad.search.index-{}-documenttype'.format(self._mode_label),\n            default=self._default_documenttype)\n\n    def __call__(self, query, max_nresults=None):\n        raise NotImplementedError\n\n    @classmethod\n    def _key_matches(cls, k, regexes):\n        \"\"\"Return which regex the key matches\n        \"\"\"\n        for regex in regexes:\n            if re.search(regex, k):\n                return regex\n\n    def show_keys(self, *args, **kwargs):\n        raise NotImplementedError(args)\n\n    def get_query(self, query):\n        \"\"\"Prepare query structure specific for a search backend.\n\n        It can also memorize within instance some parameters of the last query\n        which could be used to assist output formatting/structuring later on\n        \"\"\"\n        raise NotImplementedError\n\n    def get_nohits_msg(self):\n        \"\"\"Given what it knows, provide recommendation in the case of no hits\"\"\"\n        return \"No search hits, wrong query? \" \\\n               \"See 'datalad search --show-keys name' for known keys \" \\\n               \"and 'datalad search --help' on how to prepare your query.\"\n\n\nclass _WhooshSearch(_Search):\n    def __init__(self, ds, force_reindex=False, **kwargs):\n        super(_WhooshSearch, self).__init__(ds, **kwargs)\n\n        self.idx_obj = None\n        # where does the bunny have the eggs?\n\n        self.index_dir = opj(str(self.ds.repo.dot_git), SEARCH_INDEX_DOTGITDIR)\n        self._mk_search_index(force_reindex)\n\n    def show_keys(self, mode, regexes=None):\n        \"\"\"\n\n        Parameters\n        ----------\n        mode: {\"name\"}\n        regexes: list of regex\n          Which keys to bother working on\n        \"\"\"\n        if mode != 'name':\n            raise NotImplementedError(\n                \"ATM %s can show only names, so please use show_keys with 'name'\"\n                % self.__class__.__name__\n            )\n        for k in self.idx_obj.schema.names():\n            if regexes and not self._key_matches(k, regexes):\n                continue\n            print(u'{}'.format(k))\n\n    def get_query(self, query):\n        # parse the query string\n        self._mk_parser()\n        # for convenience we accept any number of args-words from the\n        # shell and put them together to a single string here\n        querystr = ' '.join(ensure_list(query))\n        # this gives a formal whoosh query\n        wquery = self.parser.parse(querystr)\n        return wquery\n\n    def _meta2doc(self, meta, val2str=True, schema=None):\n        raise NotImplementedError\n\n    def _mk_schema(self):\n        raise NotImplementedError\n\n    def _mk_parser(self):\n        raise NotImplementedError\n\n    def _mk_search_index(self, force_reindex):\n        \"\"\"Generic entrypoint to index generation\n\n        The actual work that determines the structure and content of the index\n        is done by functions that are passed in as arguments\n\n        `meta2doc` - must return dict for index document from result input\n        \"\"\"\n        from whoosh import index as widx\n        from .metadata import get_ds_aggregate_db_locations\n        dbloc, db_base_path = get_ds_aggregate_db_locations(self.ds)\n        # what is the lastest state of aggregated metadata\n        metadata_state = self.ds.repo.get_last_commit_hexsha(relpath(dbloc, start=self.ds.path))\n        # use location common to all index types, they would all invalidate\n        # simultaneously\n        stamp_fname = opj(self.index_dir, 'datalad_metadata_state')\n        index_dir = opj(self.index_dir, self._mode_label)\n\n        if (not force_reindex) and \\\n                exists(index_dir) and \\\n                exists(stamp_fname) and \\\n                open(stamp_fname).read() == metadata_state:\n            try:\n                # TODO check that the index schema is the same\n                # as the one we would have used for reindexing\n                # TODO support incremental re-indexing, whoosh can do it\n                idx = widx.open_dir(index_dir)\n                lgr.debug(\n                    'Search index contains %i documents',\n                    idx.doc_count())\n                self.idx_obj = idx\n                return\n            except widx.LockError as e:\n                raise e\n            except widx.IndexError as e:\n                # Generic index error.\n                # we try to regenerate\n                lgr.warning(\n                    \"Cannot open existing index %s (%s), will regenerate\",\n                    index_dir, exc_str(e)\n                )\n            except widx.IndexVersionError as e:  # (msg, version, release=None)\n                # Raised when you try to open an index using a format that the\n                # current version of Whoosh cannot read. That is, when the index\n                # you're trying to open is either not backward or forward\n                # compatible with this version of Whoosh.\n                # we try to regenerate\n                lgr.warning(exc_str(e))\n                pass\n            except widx.OutOfDateError as e:\n                # Raised when you try to commit changes to an index which is not\n                # the latest generation.\n                # this should not happen here, but if it does ... KABOOM\n                raise\n            except widx.EmptyIndexError as e:\n                # Raised when you try to work with an index that has no indexed\n                # terms.\n                # we can just continue with generating an index\n                pass\n            except ValueError as e:\n                if 'unsupported pickle protocol' in str(e):\n                    lgr.warning(\n                        \"Cannot open existing index %s (%s), will regenerate\",\n                        index_dir, exc_str(e)\n                    )\n                else:\n                    raise\n\n        lgr.info('{} search index'.format(\n            'Rebuilding' if exists(index_dir) else 'Building'))\n\n        if not exists(index_dir):\n            os.makedirs(index_dir)\n\n        # this is a pretty cheap call that just pull this info from a file\n        dsinfo = self.ds.metadata(\n            get_aggregates=True,\n            return_type='list',\n            result_renderer='disabled')\n\n        self._mk_schema(dsinfo)\n\n        idx_obj = widx.create_in(index_dir, self.schema)\n        idx = idx_obj.writer(\n            # cache size per process\n            limitmb=cfg.obtain('datalad.search.indexercachesize'),\n            # disable parallel indexing for now till #1927 is resolved\n            ## number of processes for indexing\n            #procs=multiprocessing.cpu_count(),\n            ## write separate index segments in each process for speed\n            ## asks for writer.commit(optimize=True)\n            #multisegment=True,\n        )\n\n        # load metadata of the base dataset and what it knows about all its subdatasets\n        # (recursively)\n        old_idx_size = 0\n        old_ds_rpath = ''\n        idx_size = 0\n        log_progress(\n            lgr.info,\n            'autofieldidxbuild',\n            'Start building search index',\n            total=len(dsinfo),\n            label='Building search index',\n            unit=' Datasets',\n        )\n        for res in query_aggregated_metadata(\n                reporton=self.documenttype,\n                ds=self.ds,\n                aps=[dict(path=self.ds.path, type='dataset')],\n                # MIH: I cannot see a case when we would not want recursion (within\n                # the metadata)\n                recursive=True):\n            # this assumes that files are reported after each dataset report,\n            # and after a subsequent dataset report no files for the previous\n            # dataset will be reported again\n            meta = res.get('metadata', {})\n            doc = self._meta2doc(meta)\n            admin = {\n                'type': res['type'],\n                'path': relpath(res['path'], start=self.ds.path),\n            }\n            if 'parentds' in res:\n                admin['parentds'] = relpath(res['parentds'], start=self.ds.path)\n            if admin['type'] == 'dataset':\n                if old_ds_rpath:\n                    lgr.debug(\n                        'Added %s on dataset %s',\n                        single_or_plural(\n                            'document',\n                            'documents',\n                            idx_size - old_idx_size,\n                            include_count=True),\n                        old_ds_rpath)\n                log_progress(lgr.info, 'autofieldidxbuild',\n                             'Indexed dataset at %s', old_ds_rpath,\n                             update=1, increment=True)\n                old_idx_size = idx_size\n                old_ds_rpath = admin['path']\n                admin['id'] = res.get('dsid', None)\n\n            doc.update({k: ensure_unicode(v) for k, v in admin.items()})\n            lgr.debug(\"Adding document to search index: {}\".format(doc))\n            # inject into index\n            idx.add_document(**doc)\n            idx_size += 1\n\n        if old_ds_rpath:\n            lgr.debug(\n                'Added %s on dataset %s',\n                single_or_plural(\n                    'document',\n                    'documents',\n                    idx_size - old_idx_size,\n                    include_count=True),\n                old_ds_rpath)\n\n        lgr.debug(\"Committing index\")\n        idx.commit(optimize=True)\n        log_progress(\n            lgr.info, 'autofieldidxbuild', 'Done building search index')\n\n        # \"timestamp\" the search index to allow for automatic invalidation\n        with open(stamp_fname, 'w') as f:\n            f.write(metadata_state)\n\n        lgr.info('Search index contains %i documents', idx_size)\n        self.idx_obj = idx_obj\n\n    def __call__(self, query, max_nresults=None, force_reindex=False, full_record=False):\n        if max_nresults is None:\n            # mode default\n            max_nresults = 20\n\n        with self.idx_obj.searcher() as searcher:\n            wquery = self.get_query(query)\n\n            # perform the actual search\n            hits = searcher.search(\n                wquery,\n                terms=True,\n                limit=max_nresults if max_nresults > 0 else None)\n            # report query stats\n            topstr = '{} top {}'.format(\n                max_nresults,\n                single_or_plural('match', 'matches', max_nresults)\n            )\n            lgr.info('Query completed in {} sec.{}'.format(\n                hits.runtime,\n                ' Reporting {}.'.format(\n                    ('up to ' + topstr)\n                    if max_nresults > 0\n                    else 'all matches'\n                )\n                if not hits.is_empty()\n                else ' No matches.'\n            ))\n\n            if not hits:\n                return\n\n            nhits = 0\n            annotated_hits = []\n            # annotate hits for full metadata report\n            for i, hit in enumerate(hits):\n                annotated_hit = dict(\n                    path=normpath(opj(self.ds.path, hit['path'])),\n                    query_matched={ensure_unicode(k): ensure_unicode(v)\n                                   if isinstance(v, unicode_srctypes) else v\n                                   for k, v in hit.matched_terms()},\n                    parentds=normpath(\n                        opj(self.ds.path, hit['parentds'])) if 'parentds' in hit else None,\n                    type=hit.get('type', None))\n                if not full_record:\n                    annotated_hit.update(\n                        refds=self.ds.path,\n                        action='search',\n                        status='ok',\n                        logger=lgr,\n                    )\n                    yield annotated_hit\n                else:\n                    annotated_hits.append(annotated_hit)\n                nhits += 1\n            if full_record:\n                for res in query_aggregated_metadata(\n                        # type is taken from hit record\n                        reporton=None,\n                        ds=self.ds,\n                        aps=annotated_hits,\n                        # never recursive, we have direct hits already\n                        recursive=False):\n                    res.update(\n                        refds=self.ds.path,\n                        action='search',\n                        status='ok',\n                        logger=lgr,\n                    )\n                    yield res\n\n            if max_nresults and nhits == max_nresults:\n                lgr.info(\n                    \"Reached the limit of {}, there could be more which \"\n                    \"were not reported.\".format(topstr)\n                )\n\n\nclass _BlobSearch(_WhooshSearch):\n    _mode_label = 'textblob'\n    _default_documenttype = 'datasets'\n\n    def _meta2doc(self, meta):\n        # coerce the entire flattened metadata dict into a comma-separated string\n        # that also includes the keys\n        return dict(meta=u', '.join(\n            u'{}: {}'.format(k, v)\n            for k, v in _meta2autofield_dict(\n                meta,\n                val2str=True,\n                schema=None).items()))\n\n    def _mk_schema(self, dsinfo):\n        from whoosh import fields as wf\n        from whoosh.analysis import StandardAnalyzer\n\n        # TODO support some customizable mapping to homogenize some metadata fields\n        # onto a given set of index keys\n        self.schema = wf.Schema(\n            id=wf.ID,\n            path=wf.ID(stored=True),\n            type=wf.ID(stored=True),\n            parentds=wf.ID(stored=True),\n            meta=wf.TEXT(\n                stored=False,\n                analyzer=StandardAnalyzer(minsize=2))\n        )\n\n    def _mk_parser(self):\n        from whoosh import qparser as qparse\n\n        # use whoosh default query parser for now\n        parser = qparse.QueryParser(\n            \"meta\",\n            schema=self.idx_obj.schema\n        )\n        parser.add_plugin(qparse.FuzzyTermPlugin())\n        parser.remove_plugin_class(qparse.PhrasePlugin)\n        parser.add_plugin(qparse.SequencePlugin())\n        self.parser = parser\n\n\nclass _AutofieldSearch(_WhooshSearch):\n    _mode_label = 'autofield'\n    _default_documenttype = 'all'\n\n    def _meta2doc(self, meta):\n        return _meta2autofield_dict(meta, val2str=True, schema=self.schema)\n\n    def _mk_schema(self, dsinfo):\n        from whoosh import fields as wf\n        from whoosh.analysis import SimpleAnalyzer\n\n        # haven for terms that have been found to be undefined\n        # (for faster decision-making upon next encounter)\n        # this will harvest all discovered term definitions\n        definitions = {\n            '@id': 'unique identifier of an entity',\n            # TODO make proper JSON-LD definition\n            'path': 'path name of an entity relative to the searched base dataset',\n            # TODO make proper JSON-LD definition\n            'parentds': 'path of the datasets that contains an entity',\n            # 'type' will not come from a metadata field, hence will not be detected\n            'type': 'type of a record',\n        }\n\n        schema_fields = {\n            n.lstrip('@'): wf.ID(stored=True, unique=n == '@id')\n            for n in definitions}\n\n        lgr.debug('Scanning for metadata keys')\n        # quick 1st pass over all dataset to gather the needed schema fields\n        log_progress(\n            lgr.info,\n            'idxschemabuild',\n            'Start building search schema',\n            total=len(dsinfo),\n            label='Building search schema',\n            unit=' Datasets',\n        )\n        for res in query_aggregated_metadata(\n                # XXX TODO After #2156 datasets may not necessarily carry all\n                # keys in the \"unique\" summary\n                reporton='datasets',\n                ds=self.ds,\n                aps=[dict(path=self.ds.path, type='dataset')],\n                recursive=True):\n            meta = res.get('metadata', {})\n            # no stringification of values for speed, we do not need/use the\n            # actual values at this point, only the keys\n            idxd = _meta2autofield_dict(meta, val2str=False)\n\n            for k in idxd:\n                schema_fields[k] = wf.TEXT(stored=False,\n                                           analyzer=SimpleAnalyzer())\n            log_progress(lgr.info, 'idxschemabuild',\n                         'Scanned dataset at %s', res['path'],\n                         update=1, increment=True)\n        log_progress(\n            lgr.info, 'idxschemabuild', 'Done building search schema')\n\n        self.schema = wf.Schema(**schema_fields)\n\n    def _mk_parser(self):\n        from whoosh import qparser as qparse\n\n        parser = qparse.MultifieldParser(\n            self.idx_obj.schema.names(),\n            self.idx_obj.schema)\n        # XXX: plugin is broken in Debian's whoosh 2.7.0-2, but already fixed\n        # upstream\n        parser.add_plugin(qparse.FuzzyTermPlugin())\n        parser.add_plugin(qparse.GtLtPlugin())\n        parser.add_plugin(qparse.SingleQuotePlugin())\n        # replace field defintion to allow for colons to be part of a field's name:\n        parser.replace_plugin(qparse.FieldsPlugin(expr=r\"(?P<text>[()<>.\\w]+|[*]):\"))\n        self.parser = parser\n\n\nclass _EGrepCSSearch(_Search):\n    _mode_label = 'egrepcs'\n    _default_documenttype = 'datasets'\n\n    def __init__(self, ds, **kwargs):\n        super(_EGrepCSSearch, self).__init__(ds, **kwargs)\n        self._queried_keys = None  # to be memoized by get_query\n\n    # If there were custom \"per-search engine\" options, we could expose\n    # --consider_ucn - search through unique content properties of the dataset\n    #    which might be more computationally demanding\n    def __call__(self, query, max_nresults=None, consider_ucn=False, full_record=True):\n        if max_nresults is None:\n            # no limit by default\n            max_nresults = 0\n        query = self.get_query(query)\n\n        nhits = 0\n        for res in query_aggregated_metadata(\n                reporton=self.documenttype,\n                ds=self.ds,\n                aps=[dict(path=self.ds.path, type='dataset')],\n                # MIH: I cannot see a case when we would not want recursion (within\n                # the metadata)\n                recursive=True):\n            # this assumes that files are reported after each dataset report,\n            # and after a subsequent dataset report no files for the previous\n            # dataset will be reported again\n            meta = res.get('metadata', {})\n            # produce a flattened metadata dict to search through\n            doc = _meta2autofield_dict(meta, val2str=True, consider_ucn=consider_ucn)\n            # inject a few basic properties into the dict\n            # analog to what the other modes do in their index\n            doc.update({\n                k: res[k] for k in ('@id', 'type', 'path', 'parentds')\n                if k in res})\n            # use search instead of match to not just get hits at the start of the string\n            # this will be slower, but avoids having to use actual regex syntax at the user\n            # side even for simple queries\n            # DOTALL is needed to handle multiline description fields and such, and still\n            # be able to match content coming for a later field\n            lgr.log(7, \"Querying %s among %d items\", query, len(doc))\n            t0 = time()\n            matches = {(q['query'] if isinstance(q, dict) else q, k):\n                       q['query'].search(v) if isinstance(q, dict) else q.search(v)\n                       for k, v in doc.items()\n                       for q in query\n                       if not isinstance(q, dict) or q['field'].match(k)}\n            dt = time() - t0\n            lgr.log(7, \"Finished querying in %f sec\", dt)\n            # retain what actually matched\n            matched = {k[1]: match.group() for k, match in matches.items() if match}\n            # implement AND behavior across query expressions, but OR behavior\n            # across queries matching multiple fields for a single query expression\n            # for multiple queries, this makes it consistent with a query that\n            # has no field specification\n            if matched and len(query) == len(set(k[0] for k in matches if matches[k])):\n                hit = dict(\n                    res,\n                    action='search',\n                    query_matched=matched,\n                )\n                yield hit\n                nhits += 1\n                if max_nresults and nhits == max_nresults:\n                    # report query stats\n                    topstr = '{} top {}'.format(\n                        max_nresults,\n                        single_or_plural('match', 'matches', max_nresults)\n                    )\n                    lgr.info(\n                        \"Reached the limit of {}, there could be more which \"\n                        \"were not reported.\".format(topstr)\n                    )\n                    break\n\n    def show_keys(self, mode=None, regexes=None):\n        \"\"\"\n\n        Parameters\n        ----------\n        mode: {\"name\", \"short\", \"full\"}\n        regexes: list of regex\n          Which keys to bother working on\n        \"\"\"\n        maxl = 100  # approx maximal line length for unique values in mode=short\n        # use a dict already, later we need to map to a definition\n        # meanwhile map to the values\n\n        keys = self._get_keys(mode)\n\n        for k in sorted(keys):\n            if regexes and not self._key_matches(k, regexes):\n                continue\n            if mode == 'name':\n                print(k)\n                continue\n\n            # do a bit more\n            stat = keys[k]\n            all_uvals = uvals = sorted(stat.uvals)\n\n            stat.uvals_str = ensure_unicode(\n                \"{} unique values: \".format(len(all_uvals))\n            )\n\n            if mode == 'short':\n                # show only up until we fill maxl\n                uvals_str = ''\n                uvals = []\n                for v in all_uvals:\n                    appendix = ('; ' if uvals else '') + v\n                    if len(uvals_str) + len(appendix) > maxl - len(stat.uvals_str):\n                        break\n                    uvals.append(v)\n                    uvals_str += appendix\n            elif mode == 'full':\n                pass\n            else:\n                raise ValueError(\n                    \"Unknown value for stats. Know full and short\")\n\n            stat.uvals_str += '; '.join(uvals)\n\n            if len(all_uvals) > len(uvals):\n                stat.uvals_str += \\\n                    '; +%s' % single_or_plural(\"value\", \"values\", len(all_uvals) - len(uvals), True)\n\n            print(\n                u'{k}\\n in  {stat.ndatasets} datasets\\n has {stat.uvals_str}'.format(\n                k=k, stat=stat\n            ))\n        # After #2156 datasets may not necessarily carry all\n        # keys in the \"unique\" summary\n        lgr.warning('In this search mode, the reported list of metadata keys may be incomplete')\n\n\n    def _get_keys(self, mode=None):\n        \"\"\"Return keys and their statistics if mode != 'name'.\"\"\"\n        class key_stat:\n            def __init__(self):\n                self.ndatasets = 0  # how many datasets have this field\n                self.uvals = set()\n\n        from collections import defaultdict\n        keys = defaultdict(key_stat)\n        for res in query_aggregated_metadata(\n                # XXX TODO After #2156 datasets may not necessarily carry all\n                # keys in the \"unique\" summary\n                reporton='datasets',\n                ds=self.ds,\n                aps=[dict(path=self.ds.path, type='dataset')],\n                recursive=True):\n            meta = res.get('metadata', {})\n            # inject a few basic properties into the dict\n            # analog to what the other modes do in their index\n            meta.update({\n                k: res.get(k, None) for k in ('@id', 'type', 'path', 'parentds')\n                # parentds is tricky all files will have it, but the dataset\n                # queried above might not (single dataset), let's force it in\n                if k == 'parentds' or k in res})\n\n            # no stringification of values for speed\n            idxd = _meta2autofield_dict(meta, val2str=False)\n\n            for k, kvals in idxd.items():\n                # TODO deal with conflicting definitions when available\n                keys[k].ndatasets += 1\n                if mode == 'name':\n                    continue\n                keys[k].uvals |= self.get_repr_uvalues(kvals)\n        return keys\n\n    def get_repr_uvalues(self, kvals):\n        kvals_set = set()\n        if not kvals:\n            return kvals_set\n        kvals_iter = (\n            kvals\n            if hasattr(kvals, '__iter__') and not isinstance(kvals, (str, bytes))\n            else [kvals]\n        )\n        return set(shortened_repr(x, 50) for x in kvals_iter)\n\n    def get_query(self, query):\n        query = ensure_list(query)\n        simple_fieldspec = re.compile(r\"(?P<field>\\S*?):(?P<query>.*)\")\n        quoted_fieldspec = re.compile(r\"'(?P<field>[^']+?)':(?P<query>.*)\")\n        query_rec_matches = [\n            simple_fieldspec.match(q) or\n            quoted_fieldspec.match(q) or\n            q\n            for q in query]\n        query_group_dicts_only = [\n            q.groupdict() for q in query_rec_matches if hasattr(q, 'groupdict')\n        ]\n        self._queried_keys = [\n            qgd['field']\n            for qgd in query_group_dicts_only\n            if ('field' in qgd and qgd['field'])\n        ]\n        if len(query_group_dicts_only) != len(query_rec_matches):\n            # we had a query element without field specification add\n            # None as an indicator of that\n            self._queried_keys.append(None)\n        # expand matches, compile expressions\n        query = [\n            {k: self._compile_query(v) for k, v in q.groupdict().items()}\n            if hasattr(q, 'groupdict') else self._compile_query(q)\n            for q in query_rec_matches\n        ]\n\n        # turn \"empty\" field specs into simple queries\n        # this is used to forcibly disable field-based search\n        # e.g. when searching for a value\n        query = [q['query']\n                 if isinstance(q, dict) and q['field'].pattern == '' else q\n                 for q in query]\n        return query\n\n    def _xfm_query(self, q):\n        # implement potential transformations of regex before they get compiled\n        return q\n\n    def _compile_query(self, q):\n        \"\"\"xfm and compile the query, with informative exception if query is incorrect\n        \"\"\"\n        q_xfmed = self._xfm_query(q)\n        try:\n            return re.compile(q_xfmed)\n        except re.error as exc:\n            omsg = \" (original: '%s')\" % q if q != q_xfmed else ''\n            raise ValueError(\n                \"regular expression '%s'%s is incorrect: %s\"\n                % (q_xfmed, omsg, exc)\n            )\n\n    def get_nohits_msg(self):\n        \"\"\"Given the query and performed search, provide recommendation\n\n        Quite often a key in the query is mistyped or I simply query for something\n        which is not actually known.  It requires --show-keys  run first, doing\n        visual search etc to mitigate.  Here we can analyze either all queried\n        keys are actually known, and if not known -- what would be the ones available.\n\n        Returns\n        -------\n        str\n          A sentence or a paragraph to be logged/output\n        \"\"\"\n        #\n        queried_keys = self._queried_keys[:]\n        if queried_keys and None in queried_keys:\n            queried_keys.pop(queried_keys.index(None))\n        if not queried_keys:\n            return  # No keys were queried, we are of no use here\n        known_keys = self._get_keys(mode='name')\n        unknown_keys = sorted(list(set(queried_keys).difference(known_keys)))\n        if not unknown_keys:\n            return  # again we are of no help here\n        msg = super(_EGrepCSSearch, self).get_nohits_msg()\n        msg += \" Following keys were not found in available metadata: %s. \" \\\n              % \", \".join(unknown_keys)\n        suggestions_msg = get_suggestions_msg(unknown_keys, known_keys)\n        if suggestions_msg:\n            msg += ' ' + suggestions_msg\n        return msg\n\n\nclass _EGrepSearch(_EGrepCSSearch):\n    _mode_label = 'egrep'\n    _default_documenttype = 'datasets'\n\n    def _xfm_query(self, q):\n        if q == q.lower():\n            # we have no upper case symbol in the query, go case-insensitive\n            return '(?i){}'.format(q)\n        else:\n            return q\n\n\n@build_doc\nclass Search(Interface):\n    \"\"\"Search dataset metadata\n\n    DataLad can search metadata extracted from a dataset and/or aggregated into\n    a superdataset (see the `aggregate-metadata` command). This makes it\n    possible to discover datasets, or individual files in a dataset even when\n    they are not available locally.\n\n    Ultimately DataLad metadata are a graph of linked data structures. However,\n    this command does not (yet) support queries that can exploit all\n    information stored in the metadata. At the moment the following search\n    modes are implemented that represent different trade-offs between the\n    expressiveness of a query and the computational and storage resources\n    required to execute a query.\n\n    - egrep (default)\n\n    - egrepcs [case-sensitive egrep]\n\n    - textblob\n\n    - autofield\n\n    An alternative default mode can be configured by tuning the\n    configuration variable 'datalad.search.default-mode'::\n\n      [datalad \"search\"]\n        default-mode = egrepcs\n\n    Each search mode has its own default configuration for what kind of\n    documents to query. The respective default can be changed via configuration\n    variables::\n\n      [datalad \"search\"]\n        index-<mode_name>-documenttype = (all|datasets|files)\n\n\n    *Mode: egrep/egrepcs*\n\n    These search modes are largely ignorant of the metadata structure, and\n    simply perform matching of a search pattern against a flat\n    string-representation of metadata. This is advantageous when the query is\n    simple and the metadata structure is irrelevant, or precisely known.\n    Moreover, it does not require a search index, hence results can be reported\n    without an initial latency for building a search index when the underlying\n    metadata has changed (e.g. due to a dataset update). By default, these\n    search modes only consider datasets and do not investigate records for\n    individual files for speed reasons. Search results are reported in the\n    order in which they were discovered.\n\n    Queries can make use of Python regular expression syntax\n    (https://docs.python.org/3/library/re.html). In `egrep` mode, matching is\n    case-insensitive when the query does not contain upper case characters, but\n    is case-sensitive when it does. In `egrepcs` mode, matching is always\n    case-sensitive. Expressions will match anywhere in a metadata string, not\n    only at the start.\n\n    When multiple queries are given, all queries have to match for a search hit\n    (AND behavior).\n\n    It is possible to search individual metadata key/value items by prefixing\n    the query with a metadata key name, separated by a colon (':'). The key\n    name can also be a regular expression to match multiple keys. A query match\n    happens when any value of an item with a matching key name matches the query\n    (OR behavior). See examples for more information.\n\n    Examples:\n\n      Query for (what happens to be) an author::\n\n        % datalad search haxby\n\n      Queries are case-INsensitive when the query contains no upper case characters,\n      and can be regular expressions. Use `egrepcs` mode when it is desired\n      to perform a case-sensitive lowercase match::\n\n        % datalad search --mode egrepcs halchenko.*haxby\n\n      This search mode performs NO analysis of the metadata content.  Therefore\n      queries can easily fail to match. For example, the above query implicitly\n      assumes that authors are listed in alphabetical order.  If that is the\n      case (which may or may not be true), the following query would yield NO\n      hits::\n\n        % datalad search Haxby.*Halchenko\n\n      The ``textblob`` search mode represents an alternative that is more\n      robust in such cases.\n\n      For more complex queries multiple query expressions can be provided that\n      all have to match to be considered a hit (AND behavior). This query\n      discovers all files (non-default behavior) that match 'bids.type=T1w'\n      AND 'nifti1.qform_code=scanner'::\n\n        % datalad -c datalad.search.index-egrep-documenttype=all search bids.type:T1w nifti1.qform_code:scanner\n\n      Key name selectors can also be expressions, which can be used to select\n      multiple keys or construct \"fuzzy\" queries. In such cases a query matches\n      when any item with a matching key matches the query (OR behavior).\n      However, multiple queries are always evaluated using an AND conjunction.\n      The following query extends the example above to match any files that\n      have either 'nifti1.qform_code=scanner' or 'nifti1.sform_code=scanner'::\n\n        % datalad -c datalad.search.index-egrep-documenttype=all search bids.type:T1w nifti1.(q|s)form_code:scanner\n\n    *Mode: textblob*\n\n    This search mode is very similar to the ``egrep`` mode, but with a few key\n    differences. A search index is built from the string-representation of\n    metadata records. By default, only datasets are included in this index, hence\n    the indexing is usually completed within a few seconds, even for hundreds\n    of datasets. This mode uses its own query language (not regular expressions)\n    that is similar to other search engines. It supports logical conjunctions\n    and fuzzy search terms. More information on this is available from the Whoosh\n    project (search engine implementation):\n\n      - Description of the Whoosh query language:\n        http://whoosh.readthedocs.io/en/latest/querylang.html)\n\n      - Description of a number of query language customizations that are\n        enabled in DataLad, such as, fuzzy term matching:\n        http://whoosh.readthedocs.io/en/latest/parsing.html#common-customizations\n\n    Importantly, search hits are scored and reported in order of descending\n    relevance, hence limiting the number of search results is more meaningful\n    than in the 'egrep' mode and can also reduce the query duration.\n\n    Examples:\n\n      Search for (what happens to be) two authors, regardless of the order in\n      which those names appear in the metadata::\n\n        % datalad search --mode textblob halchenko haxby\n\n      Fuzzy search when you only have an approximate idea what you are looking\n      for or how it is spelled::\n\n        % datalad search --mode textblob haxbi~\n\n      Very fuzzy search, when you are basically only confident about the first\n      two characters and how it sounds approximately (or more precisely: allow\n      for three edits and require matching of the first two characters)::\n\n        % datalad search --mode textblob haksbi~3/2\n\n      Combine fuzzy search with logical constructs::\n\n        % datalad search --mode textblob 'haxbi~ AND (hanke OR halchenko)'\n\n\n    *Mode: autofield*\n\n    This mode is similar to the 'textblob' mode, but builds a vastly more\n    detailed search index that represents individual metadata variables as\n    individual fields. By default, this search index includes records for\n    datasets and individual fields, hence it can grow very quickly into\n    a huge structure that can easily take an hour or more to build and require\n    more than a GB of storage. However, limiting it to documents on datasets\n    (see above) retains the enhanced expressiveness of queries while\n    dramatically reducing the resource demands.\n\n    Examples:\n\n      List names of search index fields (auto-discovered from the set of\n      indexed datasets) which either have a field starting with \"age\" or\n      \"gender\"::\n\n        % datalad search --mode autofield --show-keys name '\\.age' '\\.gender'\n\n      Fuzzy search for datasets with an author that is specified in a particular\n      metadata field::\n\n        % datalad search --mode autofield bids.author:haxbi~ type:dataset\n\n      Search for individual files that carry a particular description\n      prefix in their 'nifti1' metadata::\n\n        % datalad search --mode autofield nifti1.description:FSL* type:file\n\n\n    *Reporting*\n\n    Search hits are returned as standard DataLad results. On the command line\n    the '--output-format' (or '-f') option can be used to tweak results for\n    further processing.\n\n    Examples:\n\n      Format search hits as a JSON stream (one hit per line)::\n\n        % datalad -f json search haxby\n\n      Custom formatting: which terms matched the query of particular\n      results. Useful for investigating fuzzy search results::\n\n        $ datalad -f '{path}: {query_matched}' search --mode autofield bids.author:haxbi~\n    \"\"\"\n    _params_ = dict(\n        dataset=Parameter(\n            args=(\"-d\", \"--dataset\"),\n            doc=\"\"\"specify the dataset to perform the query operation on. If\n            no dataset is given, an attempt is made to identify the dataset\n            based on the current working directory and/or the `path` given\"\"\",\n            constraints=EnsureDataset() | EnsureNone()),\n        query=Parameter(\n            args=(\"query\",),\n            metavar='QUERY',\n            nargs=\"*\",\n            doc=\"\"\"query string, supported syntax and features depends on the\n            selected search mode (see documentation)\"\"\"),\n        force_reindex=Parameter(\n            args=(\"--reindex\",),\n            dest='force_reindex',\n            action='store_true',\n            doc=\"\"\"force rebuilding the search index, even if no change in the\n            dataset's state has been detected, for example, when the index\n            documenttype configuration has changed.\"\"\"),\n        max_nresults=Parameter(\n            args=(\"--max-nresults\",),\n            doc=\"\"\"maxmimum number of search results to report. Setting this\n            to 0 will report all search matches. Depending on the mode this\n            can search substantially slower. If not specified, a\n            mode-specific default setting will be used.\"\"\",\n            constraints=EnsureInt() | EnsureNone()),\n        mode=Parameter(\n            args=(\"--mode\",),\n            choices=('egrep', 'textblob', 'autofield'),\n            doc=\"\"\"Mode of search index structure and content. See section\n            SEARCH MODES for details.\"\"\"),\n        full_record=Parameter(\n            args=(\"--full-record\", '-f'),\n            action='store_true',\n            doc=\"\"\"If set, return the full metadata record for each search hit.\n            Depending on the search mode this might require additional queries.\n            By default, only data that is available to the respective search modes\n            is returned. This always includes essential information, such as the\n            path and the type.\"\"\"),\n        show_keys=Parameter(\n            args=('--show-keys',),\n            choices=('name', 'short', 'full'),\n            default=None,\n            doc=\"\"\"if given, a list of known search keys is shown. If 'name' -\n            only the name is printed one per line. If 'short' or 'full',\n            statistics (in how many datasets, and how many unique values) are\n            printed. 'short' truncates the listing of unique values.\n            QUERY, if provided, is regular expressions any of which keys should\n            contain.\n            No other action is performed (except for reindexing), even if other\n            arguments are given. Each key is accompanied by a term definition in\n            parenthesis (TODO). In most cases a definition is given in the form\n            of a URL. If an ontology definition for a term is known, this URL\n            can resolve to a webpage that provides a comprehensive definition\n            of the term. However, for speed reasons term resolution is solely done\n            on information contained in a local dataset's metadata, and definition\n            URLs might be outdated or point to no longer existing resources.\"\"\"),\n        show_query=Parameter(\n            args=('--show-query',),\n            action='store_true',\n            doc=\"\"\"if given, the formal query that was generated from the given\n            query string is shown, but not actually executed. This is mostly useful\n            for debugging purposes.\"\"\"),\n    )\n\n    @staticmethod\n    @datasetmethod(name='search')\n    @eval_results\n    def __call__(query=None,\n                 dataset=None,\n                 force_reindex=False,\n                 max_nresults=None,\n                 mode=None,\n                 full_record=False,\n                 show_keys=None,\n                 show_query=False):\n        try:\n            ds = require_dataset(dataset, check_installed=True, purpose='dataset search')\n            if ds.id is None:\n                raise NoDatasetFound(\n                    \"This does not seem to be a dataset (no DataLad dataset ID \"\n                    \"found). 'datalad create --force %s' can initialize \"\n                    \"this repository as a DataLad dataset\" % ds.path)\n        except NoDatasetFound:\n            for r in _search_from_virgin_install(dataset, query):\n                yield r\n            return\n\n        if mode is None:\n            # let's get inspired by what the dataset/user think is\n            # default\n            mode = ds.config.obtain('datalad.search.default-mode')\n\n        if mode == 'egrep':\n            searcher = _EGrepSearch\n        elif mode == 'egrepcs':\n            searcher = _EGrepCSSearch\n        elif mode == 'textblob':\n            searcher = _BlobSearch\n        elif mode == 'autofield':\n            searcher = _AutofieldSearch\n        else:\n            raise ValueError(\n                'unknown search mode \"{}\"'.format(mode))\n\n        searcher = searcher(ds, force_reindex=force_reindex)\n\n        if show_keys:\n            searcher.show_keys(show_keys, regexes=query)\n            return\n\n        if not query:\n            return\n\n        if show_query:\n            print(repr(searcher.get_query(query)))\n            return\n\n        nhits = 0\n        for r in searcher(\n                query,\n                max_nresults=max_nresults,\n                full_record=full_record):\n            nhits += 1\n            yield r\n        if not nhits:\n            lgr.info(searcher.get_nohits_msg() or 'no hits')\n",
            "header": {
                "file": {
                    "path": "datalad/metadata/search.py"
                },
                "misc_line": null
            },
            "index_line": {
                "old_commit_hash": "7698d2474",
                "new_commit_hash": "4e8977424",
                "mode": "100644"
            },
            "is_binary_file": false,
            "binary_line": null,
            "minus_file": {
                "path": "a/datalad/metadata/search.py"
            },
            "plus_file": {
                "path": "b/datalad/metadata/search.py"
            },
            "hunks": [
                {
                    "descriptor": {
                        "old_range": {
                            "start": 25,
                            "length": 8
                        },
                        "new_range": {
                            "start": 25,
                            "length": 6
                        },
                        "section": "from os.path import normpath"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "import sys",
                                "type": "context"
                            },
                            {
                                "content": "from time import time",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "from pkg_resources import EntryPoint, iter_entry_points",
                                "type": "deleted"
                            },
                            {
                                "content": "",
                                "type": "deleted"
                            },
                            {
                                "content": "from datalad import cfg",
                                "type": "context"
                            },
                            {
                                "content": "from datalad.interface.base import Interface",
                                "type": "context"
                            },
                            {
                                "content": "from datalad.interface.base import build_doc",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [],
                    "added_entities": [],
                    "deleted_entities": [
                        {
                            "file_name": "datalad/metadata/search.py",
                            "type": "statement",
                            "name": "",
                            "content": "from pkg_resources import EntryPoint, iter_entry_points",
                            "ast_type_str": "ImportFrom",
                            "start_lineno": 28,
                            "end_lineno": 28,
                            "parent": null
                        }
                    ]
                },
                {
                    "descriptor": {
                        "old_range": {
                            "start": 175,
                            "length": 7
                        },
                        "new_range": {
                            "start": 173,
                            "length": 11
                        },
                        "section": "def _meta2autofield_dict(meta, val2str=True, schema=None, consider_ucn=True):"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "            else:",
                                "type": "context"
                            },
                            {
                                "content": "                yield key, v",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "    def get_indexer_for(metadata_format_name: str) -> callable:",
                                "type": "deleted"
                            },
                            {
                                "content": "    def get_indexer(metadata_format_name: str) -> callable:",
                                "type": "added"
                            },
                            {
                                "content": "        from pkg_resources import EntryPoint, iter_entry_points",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "added"
                            },
                            {
                                "content": "        # Use the first returned entry point, if any, because there should be",
                                "type": "added"
                            },
                            {
                                "content": "        # at most one indexer for ``metadata_format_name´´.",
                                "type": "added"
                            },
                            {
                                "content": "        indexer = (tuple(iter_entry_points('datalad.metadata.indexers', metadata_format_name)) or (None,))[0]",
                                "type": "context"
                            },
                            {
                                "content": "        if isinstance(indexer, EntryPoint):",
                                "type": "context"
                            },
                            {
                                "content": "            try:",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [
                        {
                            "file_name": "datalad/metadata/search.py",
                            "type": "function",
                            "name": "_meta2autofield_dict",
                            "content": "def _meta2autofield_dict(meta, val2str=True, schema=None, consider_ucn=True):\n    \"\"\"Takes care of dtype conversion into unicode, potential key mappings\n    and concatenation of sequence-type fields into CSV strings\n\n    - if `consider_ucn` (default) it would copy keys from\n      datalad_unique_content_properties into `meta` for that extractor\n    - ... TODO ...\n    \"\"\"\n    if consider_ucn:\n        # loop over all metadata sources and the report of their unique values\n        ucnprops = meta.get(\"datalad_unique_content_properties\", {})\n        for src, umeta in ucnprops.items():\n            srcmeta = meta.get(src, {})\n            for uk in umeta:\n                if uk in srcmeta:\n                    # we have a real entry for this key in the dataset metadata\n                    # ignore any generated unique value list in favor of the\n                    # tailored data\n                    continue\n                srcmeta[uk] = _listdict2dictlist(umeta[uk], strict=False) if umeta[uk] is not None else None\n            if src not in meta and srcmeta:\n                meta[src] = srcmeta  # assign the new one back\n\n    srcmeta = None   # for paranoids to avoid some kind of manipulation of the last\n\n    def _deep_kv(basekey, dct):\n        \"\"\"Return key/value pairs of any depth following a rule for key\n        composition\n\n        dct must be a dict\n        \"\"\"\n        for k, v in dct.items():\n            if (k != '@id' and k.startswith('@')) or k == 'datalad_unique_content_properties':\n                # ignore all JSON-LD specials, but @id\n                continue\n            # TODO `k` might need remapping, if another key was already found\n            # with the same definition\n            key = u'{}{}'.format(\n                basekey,\n                # replace now special chars, and avoid spaces\n                # `os.sep` needs to go, because whoosh uses the field name for\n                # temp files during index staging, and trips over absent \"directories\"\n                # TODO maybe even kill parentheses\n                # TODO actually, it might be better to have an explicit whitelist\n                k.lstrip('@').replace(os.sep, '_').replace(' ', '_').replace('-', '_').replace('.', '_').replace(':', '-')\n            )\n            if isinstance(v, list):\n                v = _listdict2dictlist(v)\n\n            if isinstance(v, dict):\n                # dive\n                for i in _deep_kv('{}.'.format(key), v):\n                    yield i\n            else:\n                yield key, v\n\n    def get_indexer_for(metadata_format_name: str) -> callable:\n        indexer = (tuple(iter_entry_points('datalad.metadata.indexers', metadata_format_name)) or (None,))[0]\n        if isinstance(indexer, EntryPoint):\n            try:\n                indexer_object = indexer.load()(metadata_format_name)\n                return indexer_object.create_index\n            except Exception as e:\n                lgr.warning('Failed to load indexer %s: %s', indexer.name, exc_str(e))\n        return lambda metadata: _deep_kv('', metadata)\n\n    def _iterable_to_unicode_string(value, perform_operation):\n        if perform_operation:\n            if isinstance(value, (list, tuple)):\n                return u' '.join(_any2unicode(i) for i in value)\n            return _any2unicode(value)\n        return value\n\n    meta = meta or {}\n    return {\n        # Collect all meta-items which have a non-dict value type and where\n        # the key not absent in a given schema.\n        **{\n            key: _iterable_to_unicode_string(value, val2str)\n            for key, value in filter(lambda kv: not isinstance(kv[1], dict), meta.items())\n            if schema is None or key in schema\n        },\n\n        # Collect all meta-items which have a dict value type and where\n        # the key is neither 'datalad_unique_content_properties' nor absent\n        # in a given schema.\n        # These values are considered as metadata, the keys are considered to\n        # be the name of the extractor, i.e. the metadata_format_name, that created\n        # the metadata.\n        **{\n            metadata_format_name + '.' + sub_key: _iterable_to_unicode_string(sub_key_value, val2str)\n            for metadata_format_name, metadata_content in filter(\n                lambda kv: isinstance(kv[1], dict) and kv[0] != 'datalad_unique_content_properties',\n                meta.items()\n            )\n            for sub_key, sub_key_value in get_indexer_for(metadata_format_name)(metadata_content)\n            if schema is None or metadata_format_name + '.' + sub_key in schema\n        }\n    }",
                            "ast_type_str": "FunctionDef",
                            "start_lineno": 122,
                            "end_lineno": 220,
                            "parent": null
                        }
                    ],
                    "added_entities": [],
                    "deleted_entities": []
                },
                {
                    "descriptor": {
                        "old_range": {
                            "start": 185,
                            "length": 19
                        },
                        "new_range": {
                            "start": 187,
                            "length": 21
                        },
                        "section": "def _meta2autofield_dict(meta, val2str=True, schema=None, consider_ucn=True):"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "                lgr.warning('Failed to load indexer %s: %s', indexer.name, exc_str(e))",
                                "type": "context"
                            },
                            {
                                "content": "        return lambda metadata: _deep_kv('', metadata)",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "    def _iterable_to_unicode_string(value, perform_operation):",
                                "type": "deleted"
                            },
                            {
                                "content": "        if perform_operation:",
                                "type": "deleted"
                            },
                            {
                                "content": "    if val2str:",
                                "type": "added"
                            },
                            {
                                "content": "        def _val2str_helper(value):",
                                "type": "added"
                            },
                            {
                                "content": "            if isinstance(value, (list, tuple)):",
                                "type": "context"
                            },
                            {
                                "content": "                return u' '.join(_any2unicode(i) for i in value)",
                                "type": "context"
                            },
                            {
                                "content": "            return _any2unicode(value)",
                                "type": "context"
                            },
                            {
                                "content": "        return value",
                                "type": "deleted"
                            },
                            {
                                "content": "    else:",
                                "type": "added"
                            },
                            {
                                "content": "        def _val2str_helper(value):",
                                "type": "added"
                            },
                            {
                                "content": "            return value",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "    meta = meta or {}",
                                "type": "context"
                            },
                            {
                                "content": "    return {",
                                "type": "context"
                            },
                            {
                                "content": "        # Collect all meta-items which have a non-dict value type and where",
                                "type": "context"
                            },
                            {
                                "content": "        # the key not absent in a given schema.",
                                "type": "context"
                            },
                            {
                                "content": "        **{",
                                "type": "context"
                            },
                            {
                                "content": "            key: _iterable_to_unicode_string(value, val2str)",
                                "type": "deleted"
                            },
                            {
                                "content": "            key: _val2str_helper(value)",
                                "type": "added"
                            },
                            {
                                "content": "            for key, value in filter(lambda kv: not isinstance(kv[1], dict), meta.items())",
                                "type": "context"
                            },
                            {
                                "content": "            if schema is None or key in schema",
                                "type": "context"
                            },
                            {
                                "content": "        },",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [
                        {
                            "file_name": "datalad/metadata/search.py",
                            "type": "function",
                            "name": "_meta2autofield_dict",
                            "content": "def _meta2autofield_dict(meta, val2str=True, schema=None, consider_ucn=True):\n    \"\"\"Takes care of dtype conversion into unicode, potential key mappings\n    and concatenation of sequence-type fields into CSV strings\n\n    - if `consider_ucn` (default) it would copy keys from\n      datalad_unique_content_properties into `meta` for that extractor\n    - ... TODO ...\n    \"\"\"\n    if consider_ucn:\n        # loop over all metadata sources and the report of their unique values\n        ucnprops = meta.get(\"datalad_unique_content_properties\", {})\n        for src, umeta in ucnprops.items():\n            srcmeta = meta.get(src, {})\n            for uk in umeta:\n                if uk in srcmeta:\n                    # we have a real entry for this key in the dataset metadata\n                    # ignore any generated unique value list in favor of the\n                    # tailored data\n                    continue\n                srcmeta[uk] = _listdict2dictlist(umeta[uk], strict=False) if umeta[uk] is not None else None\n            if src not in meta and srcmeta:\n                meta[src] = srcmeta  # assign the new one back\n\n    srcmeta = None   # for paranoids to avoid some kind of manipulation of the last\n\n    def _deep_kv(basekey, dct):\n        \"\"\"Return key/value pairs of any depth following a rule for key\n        composition\n\n        dct must be a dict\n        \"\"\"\n        for k, v in dct.items():\n            if (k != '@id' and k.startswith('@')) or k == 'datalad_unique_content_properties':\n                # ignore all JSON-LD specials, but @id\n                continue\n            # TODO `k` might need remapping, if another key was already found\n            # with the same definition\n            key = u'{}{}'.format(\n                basekey,\n                # replace now special chars, and avoid spaces\n                # `os.sep` needs to go, because whoosh uses the field name for\n                # temp files during index staging, and trips over absent \"directories\"\n                # TODO maybe even kill parentheses\n                # TODO actually, it might be better to have an explicit whitelist\n                k.lstrip('@').replace(os.sep, '_').replace(' ', '_').replace('-', '_').replace('.', '_').replace(':', '-')\n            )\n            if isinstance(v, list):\n                v = _listdict2dictlist(v)\n\n            if isinstance(v, dict):\n                # dive\n                for i in _deep_kv('{}.'.format(key), v):\n                    yield i\n            else:\n                yield key, v\n\n    def get_indexer_for(metadata_format_name: str) -> callable:\n        indexer = (tuple(iter_entry_points('datalad.metadata.indexers', metadata_format_name)) or (None,))[0]\n        if isinstance(indexer, EntryPoint):\n            try:\n                indexer_object = indexer.load()(metadata_format_name)\n                return indexer_object.create_index\n            except Exception as e:\n                lgr.warning('Failed to load indexer %s: %s', indexer.name, exc_str(e))\n        return lambda metadata: _deep_kv('', metadata)\n\n    def _iterable_to_unicode_string(value, perform_operation):\n        if perform_operation:\n            if isinstance(value, (list, tuple)):\n                return u' '.join(_any2unicode(i) for i in value)\n            return _any2unicode(value)\n        return value\n\n    meta = meta or {}\n    return {\n        # Collect all meta-items which have a non-dict value type and where\n        # the key not absent in a given schema.\n        **{\n            key: _iterable_to_unicode_string(value, val2str)\n            for key, value in filter(lambda kv: not isinstance(kv[1], dict), meta.items())\n            if schema is None or key in schema\n        },\n\n        # Collect all meta-items which have a dict value type and where\n        # the key is neither 'datalad_unique_content_properties' nor absent\n        # in a given schema.\n        # These values are considered as metadata, the keys are considered to\n        # be the name of the extractor, i.e. the metadata_format_name, that created\n        # the metadata.\n        **{\n            metadata_format_name + '.' + sub_key: _iterable_to_unicode_string(sub_key_value, val2str)\n            for metadata_format_name, metadata_content in filter(\n                lambda kv: isinstance(kv[1], dict) and kv[0] != 'datalad_unique_content_properties',\n                meta.items()\n            )\n            for sub_key, sub_key_value in get_indexer_for(metadata_format_name)(metadata_content)\n            if schema is None or metadata_format_name + '.' + sub_key in schema\n        }\n    }",
                            "ast_type_str": "FunctionDef",
                            "start_lineno": 122,
                            "end_lineno": 220,
                            "parent": null
                        }
                    ],
                    "added_entities": [],
                    "deleted_entities": []
                },
                {
                    "descriptor": {
                        "old_range": {
                            "start": 209,
                            "length": 12
                        },
                        "new_range": {
                            "start": 213,
                            "length": 12
                        },
                        "section": "def _meta2autofield_dict(meta, val2str=True, schema=None, consider_ucn=True):"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "        # be the name of the extractor, i.e. the metadata_format_name, that created",
                                "type": "context"
                            },
                            {
                                "content": "        # the metadata.",
                                "type": "context"
                            },
                            {
                                "content": "        **{",
                                "type": "context"
                            },
                            {
                                "content": "            metadata_format_name + '.' + sub_key: _iterable_to_unicode_string(sub_key_value, val2str)",
                                "type": "deleted"
                            },
                            {
                                "content": "            metadata_format_name + '.' + sub_key: _val2str_helper(sub_key_value)",
                                "type": "added"
                            },
                            {
                                "content": "            for metadata_format_name, metadata_content in filter(",
                                "type": "context"
                            },
                            {
                                "content": "                lambda kv: isinstance(kv[1], dict) and kv[0] != 'datalad_unique_content_properties',",
                                "type": "context"
                            },
                            {
                                "content": "                meta.items()",
                                "type": "context"
                            },
                            {
                                "content": "            )",
                                "type": "context"
                            },
                            {
                                "content": "            for sub_key, sub_key_value in get_indexer_for(metadata_format_name)(metadata_content)",
                                "type": "deleted"
                            },
                            {
                                "content": "            for sub_key, sub_key_value in get_indexer(metadata_format_name)(metadata_content)",
                                "type": "added"
                            },
                            {
                                "content": "            if schema is None or metadata_format_name + '.' + sub_key in schema",
                                "type": "context"
                            },
                            {
                                "content": "        }",
                                "type": "context"
                            },
                            {
                                "content": "    }",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [
                        {
                            "file_name": "datalad/metadata/search.py",
                            "type": "function",
                            "name": "_meta2autofield_dict",
                            "content": "def _meta2autofield_dict(meta, val2str=True, schema=None, consider_ucn=True):\n    \"\"\"Takes care of dtype conversion into unicode, potential key mappings\n    and concatenation of sequence-type fields into CSV strings\n\n    - if `consider_ucn` (default) it would copy keys from\n      datalad_unique_content_properties into `meta` for that extractor\n    - ... TODO ...\n    \"\"\"\n    if consider_ucn:\n        # loop over all metadata sources and the report of their unique values\n        ucnprops = meta.get(\"datalad_unique_content_properties\", {})\n        for src, umeta in ucnprops.items():\n            srcmeta = meta.get(src, {})\n            for uk in umeta:\n                if uk in srcmeta:\n                    # we have a real entry for this key in the dataset metadata\n                    # ignore any generated unique value list in favor of the\n                    # tailored data\n                    continue\n                srcmeta[uk] = _listdict2dictlist(umeta[uk], strict=False) if umeta[uk] is not None else None\n            if src not in meta and srcmeta:\n                meta[src] = srcmeta  # assign the new one back\n\n    srcmeta = None   # for paranoids to avoid some kind of manipulation of the last\n\n    def _deep_kv(basekey, dct):\n        \"\"\"Return key/value pairs of any depth following a rule for key\n        composition\n\n        dct must be a dict\n        \"\"\"\n        for k, v in dct.items():\n            if (k != '@id' and k.startswith('@')) or k == 'datalad_unique_content_properties':\n                # ignore all JSON-LD specials, but @id\n                continue\n            # TODO `k` might need remapping, if another key was already found\n            # with the same definition\n            key = u'{}{}'.format(\n                basekey,\n                # replace now special chars, and avoid spaces\n                # `os.sep` needs to go, because whoosh uses the field name for\n                # temp files during index staging, and trips over absent \"directories\"\n                # TODO maybe even kill parentheses\n                # TODO actually, it might be better to have an explicit whitelist\n                k.lstrip('@').replace(os.sep, '_').replace(' ', '_').replace('-', '_').replace('.', '_').replace(':', '-')\n            )\n            if isinstance(v, list):\n                v = _listdict2dictlist(v)\n\n            if isinstance(v, dict):\n                # dive\n                for i in _deep_kv('{}.'.format(key), v):\n                    yield i\n            else:\n                yield key, v\n\n    def get_indexer_for(metadata_format_name: str) -> callable:\n        indexer = (tuple(iter_entry_points('datalad.metadata.indexers', metadata_format_name)) or (None,))[0]\n        if isinstance(indexer, EntryPoint):\n            try:\n                indexer_object = indexer.load()(metadata_format_name)\n                return indexer_object.create_index\n            except Exception as e:\n                lgr.warning('Failed to load indexer %s: %s', indexer.name, exc_str(e))\n        return lambda metadata: _deep_kv('', metadata)\n\n    def _iterable_to_unicode_string(value, perform_operation):\n        if perform_operation:\n            if isinstance(value, (list, tuple)):\n                return u' '.join(_any2unicode(i) for i in value)\n            return _any2unicode(value)\n        return value\n\n    meta = meta or {}\n    return {\n        # Collect all meta-items which have a non-dict value type and where\n        # the key not absent in a given schema.\n        **{\n            key: _iterable_to_unicode_string(value, val2str)\n            for key, value in filter(lambda kv: not isinstance(kv[1], dict), meta.items())\n            if schema is None or key in schema\n        },\n\n        # Collect all meta-items which have a dict value type and where\n        # the key is neither 'datalad_unique_content_properties' nor absent\n        # in a given schema.\n        # These values are considered as metadata, the keys are considered to\n        # be the name of the extractor, i.e. the metadata_format_name, that created\n        # the metadata.\n        **{\n            metadata_format_name + '.' + sub_key: _iterable_to_unicode_string(sub_key_value, val2str)\n            for metadata_format_name, metadata_content in filter(\n                lambda kv: isinstance(kv[1], dict) and kv[0] != 'datalad_unique_content_properties',\n                meta.items()\n            )\n            for sub_key, sub_key_value in get_indexer_for(metadata_format_name)(metadata_content)\n            if schema is None or metadata_format_name + '.' + sub_key in schema\n        }\n    }",
                            "ast_type_str": "FunctionDef",
                            "start_lineno": 122,
                            "end_lineno": 220,
                            "parent": null
                        }
                    ],
                    "added_entities": [],
                    "deleted_entities": []
                }
            ]
        },
        {
            "old_file_content": "# emacs: -*- mode: python-mode; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# -*- coding: utf-8 -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Some additional tests for search command (some are within test_base)\"\"\"\n\nimport logging\nfrom shutil import copy\nfrom unittest.mock import patch, MagicMock\nimport os\nfrom os import makedirs\nfrom os.path import (\n    dirname,\n    join as opj,\n)\nfrom datalad.api import Dataset\nfrom datalad.utils import (\n    chpwd,\n    swallow_logs,\n    swallow_outputs,\n)\nfrom datalad.tests.utils import (\n    assert_equal,\n    assert_in,\n    assert_re_in,\n    assert_is_generator,\n    assert_raises,\n    assert_repo_status,\n    assert_result_count,\n    eq_,\n    known_failure_githubci_win,\n    ok_file_under_git,\n    patch_config,\n    SkipTest,\n    with_tempfile,\n    with_testsui,\n)\nfrom datalad.support.exceptions import NoDatasetFound\n\nfrom datalad.api import search\n\nfrom ..search import (\n    _listdict2dictlist,\n    _meta2autofield_dict,\n)\n\n\n@with_testsui(interactive=False)\n@with_tempfile(mkdir=True)\ndef test_search_outside1_noninteractive_ui(tdir):\n    # we should raise an informative exception\n    with chpwd(tdir):\n        with assert_raises(NoDatasetFound) as cme:\n            list(search(\"bu\"))\n        assert_in('run interactively', str(cme.exception))\n\n\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\ndef test_search_outside1(tdir, newhome):\n    with chpwd(tdir):\n        # should fail since directory exists, but not a dataset\n        # should not even waste our response ;)\n        with patch_config({'datalad.locations.default-dataset': newhome}):\n            gen = search(\"bu\", return_type='generator')\n            assert_is_generator(gen)\n            assert_raises(NoDatasetFound, next, gen)\n\n        # and if we point to some non-existing dataset\n        with assert_raises(ValueError):\n            next(search(\"bu\", dataset=newhome))\n\n\n@with_testsui(responses='yes')\n@with_tempfile(mkdir=True)\n@with_tempfile()\ndef test_search_outside1_install_default_ds(tdir, default_dspath):\n    with chpwd(tdir):\n        # let's mock out even actual install/search calls\n        with \\\n            patch_config({'datalad.locations.default-dataset': default_dspath}), \\\n            patch('datalad.api.install',\n                  return_value=Dataset(default_dspath)) as mock_install, \\\n            patch('datalad.distribution.dataset.Dataset.search',\n                  new_callable=_mock_search):\n            _check_mocked_install(default_dspath, mock_install)\n\n            # now on subsequent run, we want to mock as if dataset already exists\n            # at central location and then do search again\n            from datalad.ui import ui\n            ui.add_responses('yes')\n            mock_install.reset_mock()\n            with patch(\n                    'datalad.distribution.dataset.Dataset.is_installed',\n                    True):\n                _check_mocked_install(default_dspath, mock_install)\n\n            # and what if we say \"no\" to install?\n            ui.add_responses('no')\n            mock_install.reset_mock()\n            with assert_raises(NoDatasetFound):\n                list(search(\".\"))\n\n            # and if path exists and is a valid dataset and we say \"no\"\n            Dataset(default_dspath).create()\n            ui.add_responses('no')\n            mock_install.reset_mock()\n            with assert_raises(NoDatasetFound):\n                list(search(\".\"))\n\n\n_mocked_search_results = [\n    {\n        'action': 'search',\n        'status': 'ok',\n        'path': 'ds1',  # this is wrong and must be an abspath\n        'matched': {'f': 'v'},  # this has nothing to do with the actual output\n    },\n    {\n        'action': 'search',\n        'status': 'ok',\n        'path': 'd2/ds2',  # this is wrong and must be an abspath\n        'matched': {'f1': 'v1'}  # this has nothing to do with the actual output,\n    },\n]\n\n\nclass _mock_search(object):\n    def __call__(*args, **kwargs):\n        for report in _mocked_search_results:\n            yield report\n\n\ndef _check_mocked_install(default_dspath, mock_install):\n    gen = search(\".\", return_type='generator')\n    assert_is_generator(gen)\n    # we no longer do any custom path tune up from the one returned by search\n    # so should match what search returns\n    assert_equal(\n        list(gen), [report\n                    for report in _mocked_search_results])\n    mock_install.assert_called_once_with(default_dspath, source='///')\n\n\n@with_tempfile\ndef test_search_non_dataset(tdir):\n    from datalad.support.gitrepo import GitRepo\n    GitRepo(tdir, create=True)\n    with assert_raises(NoDatasetFound) as cme:\n        list(search('smth', dataset=tdir))\n    # Should instruct user how that repo could become a datalad dataset\n    assert_in(\"datalad create --force\", str(cme.exception))\n\n\n@known_failure_githubci_win\n@with_tempfile(mkdir=True)\ndef test_within_ds_file_search(path):\n    try:\n        import mutagen\n    except ImportError:\n        raise SkipTest\n    ds = Dataset(path).create(force=True)\n    # override default and search for datasets and files for this test\n    for m in ('egrep', 'textblob', 'autofield'):\n        ds.config.add(\n            'datalad.search.index-{}-documenttype'.format(m), 'all',\n            where='dataset')\n    ds.config.add('datalad.metadata.nativetype', 'audio', where='dataset')\n    makedirs(opj(path, 'stim'))\n    for src, dst in (\n            ('audio.mp3', opj('stim', 'stim1.mp3')),):\n        copy(\n            opj(dirname(dirname(__file__)), 'tests', 'data', src),\n            opj(path, dst))\n    ds.save()\n    ok_file_under_git(path, opj('stim', 'stim1.mp3'), annexed=True)\n    # If it is not under annex, below addition of metadata silently does\n    # not do anything\n    ds.repo.set_metadata(\n        opj('stim', 'stim1.mp3'), init={'importance': 'very'})\n    ds.aggregate_metadata()\n    assert_repo_status(ds.path)\n    # basic sanity check on the metadata structure of the dataset\n    dsmeta = ds.metadata('.', reporton='datasets')[0]['metadata']\n    for src in ('audio',):\n        # something for each one\n        assert_in(src, dsmeta)\n        # each src declares its own context\n        assert_in('@context', dsmeta[src])\n        # we have a unique content metadata summary for each src\n        assert_in(src, dsmeta['datalad_unique_content_properties'])\n\n    # test default behavior\n    with swallow_outputs() as cmo:\n        ds.search(show_keys='name', mode='textblob')\n\n        assert_in(\"\"\"\\\nid\nmeta\nparentds\npath\ntype\n\"\"\", cmo.out)\n\n    target_out = \"\"\"\\\nannex.importance\nannex.key\naudio.bitrate\naudio.duration(s)\naudio.format\naudio.music-Genre\naudio.music-album\naudio.music-artist\naudio.music-channels\naudio.music-sample_rate\naudio.name\naudio.tracknumber\ndatalad_core.id\ndatalad_core.refcommit\nid\nparentds\npath\ntype\n\"\"\"\n\n    # test default behavior while limiting set of keys reported\n    with swallow_outputs() as cmo:\n        ds.search(['\\.id', 'artist$'], show_keys='short')\n        out_lines = [l for l in cmo.out.split(os.linesep) if l]\n        # test that only the ones matching were returned\n        assert_equal(\n            [l for l in out_lines if not l.startswith(' ')],\n            ['audio.music-artist', 'datalad_core.id']\n        )\n        # more specific test which would also test formatting\n        assert_equal(\n            out_lines,\n            ['audio.music-artist',\n             ' in  1 datasets', \" has 1 unique values: 'dlartist'\",\n             'datalad_core.id',\n             ' in  1 datasets',\n             # we have them sorted\n             \" has 1 unique values: '%s'\" % ds.id\n             ]\n        )\n\n    with assert_raises(ValueError) as cme:\n        ds.search('*wrong')\n    assert_re_in(\n        r\"regular expression '\\(\\?i\\)\\*wrong' \\(original: '\\*wrong'\\) is incorrect: \",\n        str(cme.exception))\n\n    # check generated autofield index keys\n    with swallow_outputs() as cmo:\n        ds.search(mode='autofield', show_keys='name')\n        # it is impossible to assess what is different from that dump\n        assert_in(target_out, cmo.out)\n\n    assert_result_count(ds.search('blablob#'), 0)\n    # now check that we can discover things from the aggregated metadata\n    for mode, query, hitpath, matched in (\n        ('egrep',\n         ':mp3',\n         opj('stim', 'stim1.mp3'),\n         {'audio.format': 'mp3'}),\n        # same as above, leading : is stripped, in indicates \"ALL FIELDS\"\n        ('egrep',\n         'mp3',\n         opj('stim', 'stim1.mp3'),\n         {'audio.format': 'mp3'}),\n        # same as above, but with AND condition\n        # get both matches\n        ('egrep',\n         ['mp3', 'type:file'],\n         opj('stim', 'stim1.mp3'),\n         {'type': 'file', 'audio.format': 'mp3'}),\n        # case insensitive search\n        ('egrep',\n         'mp3',\n         opj('stim', 'stim1.mp3'),\n         {'audio.format': 'mp3'}),\n        # field selection by expression\n        ('egrep',\n         'audio\\.+:mp3',\n         opj('stim', 'stim1.mp3'),\n         {'audio.format': 'mp3'}),\n        # random keyword query\n        ('textblob',\n         'mp3',\n         opj('stim', 'stim1.mp3'),\n         {'meta': 'mp3'}),\n        # report which field matched with auto-field\n        ('autofield',\n         'mp3',\n         opj('stim', 'stim1.mp3'),\n         {'audio.format': 'mp3'}),\n        # XXX next one is not supported by current text field analyser\n        # decomposes the mime type in [mime, audio, mp3]\n        # ('autofield',\n        # \"'mime:audio/mp3'\",\n        # opj('stim', 'stim1.mp3'),\n        # 'audio.format', 'mime:audio/mp3'),\n        # but this one works\n        ('autofield',\n         \"'mime audio mp3'\",\n         opj('stim', 'stim1.mp3'),\n         {'audio.format': 'mp3'}),\n        # TODO extend with more complex queries to test whoosh\n        # query language configuration\n    ):\n        res = ds.search(query, mode=mode, full_record=True)\n        assert_result_count(\n            res, 1, type='file', path=opj(ds.path, hitpath),\n            # each file must report the ID of the dataset it is from, critical for\n            # discovering related content\n            dsid=ds.id)\n        # in egrep we currently do not search unique values\n        # and the queries above aim at files\n        assert_result_count(res, 1 if mode == 'egrep' else 2)\n        if mode != 'egrep':\n            assert_result_count(\n                res, 1, type='dataset', path=ds.path, dsid=ds.id)\n        # test the key and specific value of the match\n        for matched_key, matched_val in matched.items():\n            assert_in(matched_key, res[-1]['query_matched'])\n            assert_equal(res[-1]['query_matched'][matched_key], matched_val)\n\n    # test a suggestion msg being logged if no hits and key is a bit off\n    with swallow_logs(new_level=logging.INFO) as cml:\n        res = ds.search('audio.formats:mp3 audio.bitsrate:1', mode='egrep')\n        assert not res\n        assert_in('Did you mean any of', cml.out)\n        assert_in('audio.format', cml.out)\n        assert_in('audio.bitrate', cml.out)\n\n\ndef test_listdict2dictlist():\n    f = _listdict2dictlist\n    l1 = [1, 3, [1, 'a']]\n    assert f(l1) is l1, \"we return it as is if no emb dict\"\n    eq_(f([{1: 2}]), {1: 2})  # inside out no need for a list\n    # inside out, join into the list, skip entry with a list, or space\n    eq_(f([{1: [2, 3], 'a': 1}, {'a': 2, 'c': ''}]), {'a': [1, 2]})\n\n\ndef test_meta2autofield_dict():\n    # Just a test that we would obtain the value stored for that extractor\n    # instead of what unique values it already had (whatever that means)\n    eq_(\n        _meta2autofield_dict({\n            'datalad_unique_content_properties':\n                {'extr1': {\"prop1\": \"v1\"}},\n            'extr1': {'prop1': 'value'}}),\n        {'extr1.prop1': 'value'}\n    )\n\n\ndef test_external_indexer():\n    \"\"\" check that external indexer are called \"\"\"\n    from pkg_resources import EntryPoint\n\n    class MockedIndexer:\n        def __init__(self, metadata_format_name: str):\n            pass\n\n        def create_index(self, metadata):\n            yield from {\n                \"jubel\": 1,\n                \"trubel\": 2\n            }.items()\n\n    class MockedEntryPoint(EntryPoint):\n        def __init__(self):\n            pass\n\n        def load(self, *args):\n            return MockedIndexer\n\n    def _mocked_iter_entry_points(group, metadata):\n        yield MockedEntryPoint()\n\n    with patch('datalad.metadata.search.iter_entry_points',\n               MagicMock(side_effect=_mocked_iter_entry_points)):\n        index = _meta2autofield_dict({\n            'datalad_unique_content_properties': {\n                'extr1': {\n                    \"prop1\": \"v1\"\n                }\n            },\n            'extr1': {\n                'prop1': 'value'\n            }\n        })\n\n    eq_(\n        index,\n        {\n            'extr1.jubel': '1',\n            'extr1.trubel': '2'\n        }\n    )\n\n\ndef test_faulty_external_indexer():\n    \"\"\" check that generic indexer is called on external indexer faults \"\"\"\n    from pkg_resources import EntryPoint\n\n    class MockedEntryPoint(EntryPoint):\n        def __init__(self):\n            self.name = 'MockedEntryPoint'\n\n        def load(self, *args):\n            raise Exception('Mocked indexer error')\n\n    def _mocked_iter_entry_points(group, metadata):\n        yield MockedEntryPoint()\n\n    with patch('datalad.metadata.search.iter_entry_points',\n               MagicMock(side_effect=_mocked_iter_entry_points)):\n        index = _meta2autofield_dict({\n            'datalad_unique_content_properties': {\n                'extr1': {\n                    \"prop1\": \"v1\"\n                }\n            },\n            'extr1': {\n                'prop1': 'value'\n            }\n        })\n\n    eq_(\n        index,\n        {\n            'extr1.prop1': 'value'\n        }\n    )\n",
            "new_file_content": "# emacs: -*- mode: python-mode; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# -*- coding: utf-8 -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Some additional tests for search command (some are within test_base)\"\"\"\n\nimport logging\nfrom shutil import copy\nfrom unittest.mock import patch, MagicMock\nimport os\nfrom os import makedirs\nfrom os.path import (\n    dirname,\n    join as opj,\n)\n\nfrom pkg_resources import EntryPoint\n\nfrom datalad.api import Dataset\nfrom datalad.utils import (\n    chpwd,\n    swallow_logs,\n    swallow_outputs,\n)\nfrom datalad.tests.utils import (\n    assert_equal,\n    assert_in,\n    assert_re_in,\n    assert_is_generator,\n    assert_raises,\n    assert_repo_status,\n    assert_result_count,\n    eq_,\n    known_failure_githubci_win,\n    ok_file_under_git,\n    patch_config,\n    SkipTest,\n    with_tempfile,\n    with_testsui,\n)\nfrom datalad.support.exceptions import NoDatasetFound\n\nfrom datalad.api import search\n\nfrom ..search import (\n    _listdict2dictlist,\n    _meta2autofield_dict,\n)\n\n\n@with_testsui(interactive=False)\n@with_tempfile(mkdir=True)\ndef test_search_outside1_noninteractive_ui(tdir):\n    # we should raise an informative exception\n    with chpwd(tdir):\n        with assert_raises(NoDatasetFound) as cme:\n            list(search(\"bu\"))\n        assert_in('run interactively', str(cme.exception))\n\n\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\ndef test_search_outside1(tdir, newhome):\n    with chpwd(tdir):\n        # should fail since directory exists, but not a dataset\n        # should not even waste our response ;)\n        with patch_config({'datalad.locations.default-dataset': newhome}):\n            gen = search(\"bu\", return_type='generator')\n            assert_is_generator(gen)\n            assert_raises(NoDatasetFound, next, gen)\n\n        # and if we point to some non-existing dataset\n        with assert_raises(ValueError):\n            next(search(\"bu\", dataset=newhome))\n\n\n@with_testsui(responses='yes')\n@with_tempfile(mkdir=True)\n@with_tempfile()\ndef test_search_outside1_install_default_ds(tdir, default_dspath):\n    with chpwd(tdir):\n        # let's mock out even actual install/search calls\n        with \\\n            patch_config({'datalad.locations.default-dataset': default_dspath}), \\\n            patch('datalad.api.install',\n                  return_value=Dataset(default_dspath)) as mock_install, \\\n            patch('datalad.distribution.dataset.Dataset.search',\n                  new_callable=_mock_search):\n            _check_mocked_install(default_dspath, mock_install)\n\n            # now on subsequent run, we want to mock as if dataset already exists\n            # at central location and then do search again\n            from datalad.ui import ui\n            ui.add_responses('yes')\n            mock_install.reset_mock()\n            with patch(\n                    'datalad.distribution.dataset.Dataset.is_installed',\n                    True):\n                _check_mocked_install(default_dspath, mock_install)\n\n            # and what if we say \"no\" to install?\n            ui.add_responses('no')\n            mock_install.reset_mock()\n            with assert_raises(NoDatasetFound):\n                list(search(\".\"))\n\n            # and if path exists and is a valid dataset and we say \"no\"\n            Dataset(default_dspath).create()\n            ui.add_responses('no')\n            mock_install.reset_mock()\n            with assert_raises(NoDatasetFound):\n                list(search(\".\"))\n\n\n_mocked_search_results = [\n    {\n        'action': 'search',\n        'status': 'ok',\n        'path': 'ds1',  # this is wrong and must be an abspath\n        'matched': {'f': 'v'},  # this has nothing to do with the actual output\n    },\n    {\n        'action': 'search',\n        'status': 'ok',\n        'path': 'd2/ds2',  # this is wrong and must be an abspath\n        'matched': {'f1': 'v1'}  # this has nothing to do with the actual output,\n    },\n]\n\n\nclass _mock_search(object):\n    def __call__(*args, **kwargs):\n        for report in _mocked_search_results:\n            yield report\n\n\ndef _check_mocked_install(default_dspath, mock_install):\n    gen = search(\".\", return_type='generator')\n    assert_is_generator(gen)\n    # we no longer do any custom path tune up from the one returned by search\n    # so should match what search returns\n    assert_equal(\n        list(gen), [report\n                    for report in _mocked_search_results])\n    mock_install.assert_called_once_with(default_dspath, source='///')\n\n\n@with_tempfile\ndef test_search_non_dataset(tdir):\n    from datalad.support.gitrepo import GitRepo\n    GitRepo(tdir, create=True)\n    with assert_raises(NoDatasetFound) as cme:\n        list(search('smth', dataset=tdir))\n    # Should instruct user how that repo could become a datalad dataset\n    assert_in(\"datalad create --force\", str(cme.exception))\n\n\n@known_failure_githubci_win\n@with_tempfile(mkdir=True)\ndef test_within_ds_file_search(path):\n    try:\n        import mutagen\n    except ImportError:\n        raise SkipTest\n    ds = Dataset(path).create(force=True)\n    # override default and search for datasets and files for this test\n    for m in ('egrep', 'textblob', 'autofield'):\n        ds.config.add(\n            'datalad.search.index-{}-documenttype'.format(m), 'all',\n            where='dataset')\n    ds.config.add('datalad.metadata.nativetype', 'audio', where='dataset')\n    makedirs(opj(path, 'stim'))\n    for src, dst in (\n            ('audio.mp3', opj('stim', 'stim1.mp3')),):\n        copy(\n            opj(dirname(dirname(__file__)), 'tests', 'data', src),\n            opj(path, dst))\n    ds.save()\n    ok_file_under_git(path, opj('stim', 'stim1.mp3'), annexed=True)\n    # If it is not under annex, below addition of metadata silently does\n    # not do anything\n    ds.repo.set_metadata(\n        opj('stim', 'stim1.mp3'), init={'importance': 'very'})\n    ds.aggregate_metadata()\n    assert_repo_status(ds.path)\n    # basic sanity check on the metadata structure of the dataset\n    dsmeta = ds.metadata('.', reporton='datasets')[0]['metadata']\n    for src in ('audio',):\n        # something for each one\n        assert_in(src, dsmeta)\n        # each src declares its own context\n        assert_in('@context', dsmeta[src])\n        # we have a unique content metadata summary for each src\n        assert_in(src, dsmeta['datalad_unique_content_properties'])\n\n    # test default behavior\n    with swallow_outputs() as cmo:\n        ds.search(show_keys='name', mode='textblob')\n\n        assert_in(\"\"\"\\\nid\nmeta\nparentds\npath\ntype\n\"\"\", cmo.out)\n\n    target_out = \"\"\"\\\nannex.importance\nannex.key\naudio.bitrate\naudio.duration(s)\naudio.format\naudio.music-Genre\naudio.music-album\naudio.music-artist\naudio.music-channels\naudio.music-sample_rate\naudio.name\naudio.tracknumber\ndatalad_core.id\ndatalad_core.refcommit\nid\nparentds\npath\ntype\n\"\"\"\n\n    # test default behavior while limiting set of keys reported\n    with swallow_outputs() as cmo:\n        ds.search(['\\.id', 'artist$'], show_keys='short')\n        out_lines = [l for l in cmo.out.split(os.linesep) if l]\n        # test that only the ones matching were returned\n        assert_equal(\n            [l for l in out_lines if not l.startswith(' ')],\n            ['audio.music-artist', 'datalad_core.id']\n        )\n        # more specific test which would also test formatting\n        assert_equal(\n            out_lines,\n            ['audio.music-artist',\n             ' in  1 datasets', \" has 1 unique values: 'dlartist'\",\n             'datalad_core.id',\n             ' in  1 datasets',\n             # we have them sorted\n             \" has 1 unique values: '%s'\" % ds.id\n             ]\n        )\n\n    with assert_raises(ValueError) as cme:\n        ds.search('*wrong')\n    assert_re_in(\n        r\"regular expression '\\(\\?i\\)\\*wrong' \\(original: '\\*wrong'\\) is incorrect: \",\n        str(cme.exception))\n\n    # check generated autofield index keys\n    with swallow_outputs() as cmo:\n        ds.search(mode='autofield', show_keys='name')\n        # it is impossible to assess what is different from that dump\n        assert_in(target_out, cmo.out)\n\n    assert_result_count(ds.search('blablob#'), 0)\n    # now check that we can discover things from the aggregated metadata\n    for mode, query, hitpath, matched in (\n        ('egrep',\n         ':mp3',\n         opj('stim', 'stim1.mp3'),\n         {'audio.format': 'mp3'}),\n        # same as above, leading : is stripped, in indicates \"ALL FIELDS\"\n        ('egrep',\n         'mp3',\n         opj('stim', 'stim1.mp3'),\n         {'audio.format': 'mp3'}),\n        # same as above, but with AND condition\n        # get both matches\n        ('egrep',\n         ['mp3', 'type:file'],\n         opj('stim', 'stim1.mp3'),\n         {'type': 'file', 'audio.format': 'mp3'}),\n        # case insensitive search\n        ('egrep',\n         'mp3',\n         opj('stim', 'stim1.mp3'),\n         {'audio.format': 'mp3'}),\n        # field selection by expression\n        ('egrep',\n         'audio\\.+:mp3',\n         opj('stim', 'stim1.mp3'),\n         {'audio.format': 'mp3'}),\n        # random keyword query\n        ('textblob',\n         'mp3',\n         opj('stim', 'stim1.mp3'),\n         {'meta': 'mp3'}),\n        # report which field matched with auto-field\n        ('autofield',\n         'mp3',\n         opj('stim', 'stim1.mp3'),\n         {'audio.format': 'mp3'}),\n        # XXX next one is not supported by current text field analyser\n        # decomposes the mime type in [mime, audio, mp3]\n        # ('autofield',\n        # \"'mime:audio/mp3'\",\n        # opj('stim', 'stim1.mp3'),\n        # 'audio.format', 'mime:audio/mp3'),\n        # but this one works\n        ('autofield',\n         \"'mime audio mp3'\",\n         opj('stim', 'stim1.mp3'),\n         {'audio.format': 'mp3'}),\n        # TODO extend with more complex queries to test whoosh\n        # query language configuration\n    ):\n        res = ds.search(query, mode=mode, full_record=True)\n        assert_result_count(\n            res, 1, type='file', path=opj(ds.path, hitpath),\n            # each file must report the ID of the dataset it is from, critical for\n            # discovering related content\n            dsid=ds.id)\n        # in egrep we currently do not search unique values\n        # and the queries above aim at files\n        assert_result_count(res, 1 if mode == 'egrep' else 2)\n        if mode != 'egrep':\n            assert_result_count(\n                res, 1, type='dataset', path=ds.path, dsid=ds.id)\n        # test the key and specific value of the match\n        for matched_key, matched_val in matched.items():\n            assert_in(matched_key, res[-1]['query_matched'])\n            assert_equal(res[-1]['query_matched'][matched_key], matched_val)\n\n    # test a suggestion msg being logged if no hits and key is a bit off\n    with swallow_logs(new_level=logging.INFO) as cml:\n        res = ds.search('audio.formats:mp3 audio.bitsrate:1', mode='egrep')\n        assert not res\n        assert_in('Did you mean any of', cml.out)\n        assert_in('audio.format', cml.out)\n        assert_in('audio.bitrate', cml.out)\n\n\ndef test_listdict2dictlist():\n    f = _listdict2dictlist\n    l1 = [1, 3, [1, 'a']]\n    assert f(l1) is l1, \"we return it as is if no emb dict\"\n    eq_(f([{1: 2}]), {1: 2})  # inside out no need for a list\n    # inside out, join into the list, skip entry with a list, or space\n    eq_(f([{1: [2, 3], 'a': 1}, {'a': 2, 'c': ''}]), {'a': [1, 2]})\n\n\ndef test_meta2autofield_dict():\n    # Just a test that we would obtain the value stored for that extractor\n    # instead of what unique values it already had (whatever that means)\n    eq_(\n        _meta2autofield_dict({\n            'datalad_unique_content_properties':\n                {'extr1': {\"prop1\": \"v1\"}},\n            'extr1': {'prop1': 'value'}}),\n        {'extr1.prop1': 'value'}\n    )\n\n\ndef test_external_indexer():\n    \"\"\" check that external indexer are called \"\"\"\n    class MockedIndexer:\n        def __init__(self, metadata_format_name: str):\n            pass\n\n        def create_index(self, metadata):\n            yield from {\n                \"jubel\": 1,\n                \"trubel\": 2\n            }.items()\n\n    class MockedEntryPoint(EntryPoint):\n        def __init__(self):\n            pass\n\n        def load(self, *args):\n            return MockedIndexer\n\n    def _mocked_iter_entry_points(group, metadata):\n        yield MockedEntryPoint()\n\n    with patch('pkg_resources.iter_entry_points',\n               MagicMock(side_effect=_mocked_iter_entry_points)):\n        index = _meta2autofield_dict({\n            'datalad_unique_content_properties': {\n                'extr1': {\n                    \"prop1\": \"v1\"\n                }\n            },\n            'extr1': {\n                'prop1': 'value'\n            }\n        })\n\n    eq_(\n        index,\n        {\n            'extr1.jubel': '1',\n            'extr1.trubel': '2'\n        }\n    )\n\n\ndef test_faulty_external_indexer():\n    \"\"\" check that generic indexer is called on external indexer faults \"\"\"\n    class MockedEntryPoint(EntryPoint):\n        def __init__(self):\n            self.name = 'MockedEntryPoint'\n\n        def load(self, *args):\n            raise Exception('Mocked indexer error')\n\n    def _mocked_iter_entry_points(group, metadata):\n        yield MockedEntryPoint()\n\n    with patch('pkg_resources.iter_entry_points',\n               MagicMock(side_effect=_mocked_iter_entry_points)):\n        index = _meta2autofield_dict({\n            'datalad_unique_content_properties': {\n                'extr1': {\n                    \"prop1\": \"v1\"\n                }\n            },\n            'extr1': {\n                'prop1': 'value'\n            }\n        })\n\n    eq_(\n        index,\n        {\n            'extr1.prop1': 'value'\n        }\n    )\n",
            "header": {
                "file": {
                    "path": "datalad/metadata/tests/test_search.py"
                },
                "misc_line": null
            },
            "index_line": {
                "old_commit_hash": "54b89d28b",
                "new_commit_hash": "3aea421f1",
                "mode": "100644"
            },
            "is_binary_file": false,
            "binary_line": null,
            "minus_file": {
                "path": "a/datalad/metadata/tests/test_search.py"
            },
            "plus_file": {
                "path": "b/datalad/metadata/tests/test_search.py"
            },
            "hunks": [
                {
                    "descriptor": {
                        "old_range": {
                            "start": 18,
                            "length": 6
                        },
                        "new_range": {
                            "start": 18,
                            "length": 9
                        },
                        "section": "from os.path import ("
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "    dirname,",
                                "type": "context"
                            },
                            {
                                "content": "    join as opj,",
                                "type": "context"
                            },
                            {
                                "content": ")",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "added"
                            },
                            {
                                "content": "from pkg_resources import EntryPoint",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "added"
                            },
                            {
                                "content": "from datalad.api import Dataset",
                                "type": "context"
                            },
                            {
                                "content": "from datalad.utils import (",
                                "type": "context"
                            },
                            {
                                "content": "    chpwd,",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [],
                    "added_entities": [
                        {
                            "file_name": "datalad/metadata/tests/test_search.py",
                            "type": "statement",
                            "name": "",
                            "content": "from pkg_resources import EntryPoint",
                            "ast_type_str": "ImportFrom",
                            "start_lineno": 22,
                            "end_lineno": 22,
                            "parent": null
                        }
                    ],
                    "deleted_entities": []
                },
                {
                    "descriptor": {
                        "old_range": {
                            "start": 362,
                            "length": 8
                        },
                        "new_range": {
                            "start": 365,
                            "length": 6
                        },
                        "section": "def test_meta2autofield_dict():"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "def test_external_indexer():",
                                "type": "context"
                            },
                            {
                                "content": "    \"\"\" check that external indexer are called \"\"\"",
                                "type": "context"
                            },
                            {
                                "content": "    from pkg_resources import EntryPoint",
                                "type": "deleted"
                            },
                            {
                                "content": "",
                                "type": "deleted"
                            },
                            {
                                "content": "    class MockedIndexer:",
                                "type": "context"
                            },
                            {
                                "content": "        def __init__(self, metadata_format_name: str):",
                                "type": "context"
                            },
                            {
                                "content": "            pass",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [
                        {
                            "file_name": "datalad/metadata/tests/test_search.py",
                            "type": "function",
                            "name": "test_external_indexer",
                            "content": "def test_external_indexer():\n    \"\"\" check that external indexer are called \"\"\"\n    from pkg_resources import EntryPoint\n\n    class MockedIndexer:\n        def __init__(self, metadata_format_name: str):\n            pass\n\n        def create_index(self, metadata):\n            yield from {\n                \"jubel\": 1,\n                \"trubel\": 2\n            }.items()\n\n    class MockedEntryPoint(EntryPoint):\n        def __init__(self):\n            pass\n\n        def load(self, *args):\n            return MockedIndexer\n\n    def _mocked_iter_entry_points(group, metadata):\n        yield MockedEntryPoint()\n\n    with patch('datalad.metadata.search.iter_entry_points',\n               MagicMock(side_effect=_mocked_iter_entry_points)):\n        index = _meta2autofield_dict({\n            'datalad_unique_content_properties': {\n                'extr1': {\n                    \"prop1\": \"v1\"\n                }\n            },\n            'extr1': {\n                'prop1': 'value'\n            }\n        })\n\n    eq_(\n        index,\n        {\n            'extr1.jubel': '1',\n            'extr1.trubel': '2'\n        }\n    )",
                            "ast_type_str": "FunctionDef",
                            "start_lineno": 363,
                            "end_lineno": 406,
                            "parent": null
                        }
                    ],
                    "added_entities": [],
                    "deleted_entities": []
                },
                {
                    "descriptor": {
                        "old_range": {
                            "start": 384,
                            "length": 7
                        },
                        "new_range": {
                            "start": 385,
                            "length": 7
                        },
                        "section": "def test_external_indexer():"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "    def _mocked_iter_entry_points(group, metadata):",
                                "type": "context"
                            },
                            {
                                "content": "        yield MockedEntryPoint()",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "    with patch('datalad.metadata.search.iter_entry_points',",
                                "type": "deleted"
                            },
                            {
                                "content": "    with patch('pkg_resources.iter_entry_points',",
                                "type": "added"
                            },
                            {
                                "content": "               MagicMock(side_effect=_mocked_iter_entry_points)):",
                                "type": "context"
                            },
                            {
                                "content": "        index = _meta2autofield_dict({",
                                "type": "context"
                            },
                            {
                                "content": "            'datalad_unique_content_properties': {",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [
                        {
                            "file_name": "datalad/metadata/tests/test_search.py",
                            "type": "function",
                            "name": "test_external_indexer",
                            "content": "def test_external_indexer():\n    \"\"\" check that external indexer are called \"\"\"\n    from pkg_resources import EntryPoint\n\n    class MockedIndexer:\n        def __init__(self, metadata_format_name: str):\n            pass\n\n        def create_index(self, metadata):\n            yield from {\n                \"jubel\": 1,\n                \"trubel\": 2\n            }.items()\n\n    class MockedEntryPoint(EntryPoint):\n        def __init__(self):\n            pass\n\n        def load(self, *args):\n            return MockedIndexer\n\n    def _mocked_iter_entry_points(group, metadata):\n        yield MockedEntryPoint()\n\n    with patch('datalad.metadata.search.iter_entry_points',\n               MagicMock(side_effect=_mocked_iter_entry_points)):\n        index = _meta2autofield_dict({\n            'datalad_unique_content_properties': {\n                'extr1': {\n                    \"prop1\": \"v1\"\n                }\n            },\n            'extr1': {\n                'prop1': 'value'\n            }\n        })\n\n    eq_(\n        index,\n        {\n            'extr1.jubel': '1',\n            'extr1.trubel': '2'\n        }\n    )",
                            "ast_type_str": "FunctionDef",
                            "start_lineno": 363,
                            "end_lineno": 406,
                            "parent": null
                        }
                    ],
                    "added_entities": [],
                    "deleted_entities": []
                },
                {
                    "descriptor": {
                        "old_range": {
                            "start": 408,
                            "length": 8
                        },
                        "new_range": {
                            "start": 409,
                            "length": 6
                        },
                        "section": "def test_external_indexer():"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "def test_faulty_external_indexer():",
                                "type": "context"
                            },
                            {
                                "content": "    \"\"\" check that generic indexer is called on external indexer faults \"\"\"",
                                "type": "context"
                            },
                            {
                                "content": "    from pkg_resources import EntryPoint",
                                "type": "deleted"
                            },
                            {
                                "content": "",
                                "type": "deleted"
                            },
                            {
                                "content": "    class MockedEntryPoint(EntryPoint):",
                                "type": "context"
                            },
                            {
                                "content": "        def __init__(self):",
                                "type": "context"
                            },
                            {
                                "content": "            self.name = 'MockedEntryPoint'",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [
                        {
                            "file_name": "datalad/metadata/tests/test_search.py",
                            "type": "function",
                            "name": "test_faulty_external_indexer",
                            "content": "def test_faulty_external_indexer():\n    \"\"\" check that generic indexer is called on external indexer faults \"\"\"\n    from pkg_resources import EntryPoint\n\n    class MockedEntryPoint(EntryPoint):\n        def __init__(self):\n            self.name = 'MockedEntryPoint'\n\n        def load(self, *args):\n            raise Exception('Mocked indexer error')\n\n    def _mocked_iter_entry_points(group, metadata):\n        yield MockedEntryPoint()\n\n    with patch('datalad.metadata.search.iter_entry_points',\n               MagicMock(side_effect=_mocked_iter_entry_points)):\n        index = _meta2autofield_dict({\n            'datalad_unique_content_properties': {\n                'extr1': {\n                    \"prop1\": \"v1\"\n                }\n            },\n            'extr1': {\n                'prop1': 'value'\n            }\n        })\n\n    eq_(\n        index,\n        {\n            'extr1.prop1': 'value'\n        }\n    )",
                            "ast_type_str": "FunctionDef",
                            "start_lineno": 409,
                            "end_lineno": 441,
                            "parent": null
                        }
                    ],
                    "added_entities": [],
                    "deleted_entities": []
                },
                {
                    "descriptor": {
                        "old_range": {
                            "start": 420,
                            "length": 7
                        },
                        "new_range": {
                            "start": 419,
                            "length": 7
                        },
                        "section": "def test_faulty_external_indexer():"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "    def _mocked_iter_entry_points(group, metadata):",
                                "type": "context"
                            },
                            {
                                "content": "        yield MockedEntryPoint()",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "    with patch('datalad.metadata.search.iter_entry_points',",
                                "type": "deleted"
                            },
                            {
                                "content": "    with patch('pkg_resources.iter_entry_points',",
                                "type": "added"
                            },
                            {
                                "content": "               MagicMock(side_effect=_mocked_iter_entry_points)):",
                                "type": "context"
                            },
                            {
                                "content": "        index = _meta2autofield_dict({",
                                "type": "context"
                            },
                            {
                                "content": "            'datalad_unique_content_properties': {",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [
                        {
                            "file_name": "datalad/metadata/tests/test_search.py",
                            "type": "function",
                            "name": "test_faulty_external_indexer",
                            "content": "def test_faulty_external_indexer():\n    \"\"\" check that generic indexer is called on external indexer faults \"\"\"\n    from pkg_resources import EntryPoint\n\n    class MockedEntryPoint(EntryPoint):\n        def __init__(self):\n            self.name = 'MockedEntryPoint'\n\n        def load(self, *args):\n            raise Exception('Mocked indexer error')\n\n    def _mocked_iter_entry_points(group, metadata):\n        yield MockedEntryPoint()\n\n    with patch('datalad.metadata.search.iter_entry_points',\n               MagicMock(side_effect=_mocked_iter_entry_points)):\n        index = _meta2autofield_dict({\n            'datalad_unique_content_properties': {\n                'extr1': {\n                    \"prop1\": \"v1\"\n                }\n            },\n            'extr1': {\n                'prop1': 'value'\n            }\n        })\n\n    eq_(\n        index,\n        {\n            'extr1.prop1': 'value'\n        }\n    )",
                            "ast_type_str": "FunctionDef",
                            "start_lineno": 409,
                            "end_lineno": 441,
                            "parent": null
                        }
                    ],
                    "added_entities": [],
                    "deleted_entities": []
                }
            ]
        }
    ],
    "old_commit_hash": "d1a45eaa510354812258cd022388cd202bff9080^",
    "new_commit_hash": "d1a45eaa510354812258cd022388cd202bff9080",
    "commit_message": "RF: apply reviewer suggestions",
    "commit_date": "2020-10-13T12:35:38+02:00",
    "metadata": {}
}
{
    "repo_name": "datalad",
    "new_commit_hash": "d1a45eaa510354812258cd022388cd202bff9080",
    "test_file_codes": [
        "# emacs: -*- mode: python-mode; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# -*- coding: utf-8 -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Some additional tests for search command (some are within test_base)\"\"\"\n\nimport logging\nfrom shutil import copy\nfrom unittest.mock import patch, MagicMock\nimport os\nfrom os import makedirs\nfrom os.path import (\n    dirname,\n    join as opj,\n)\n\nfrom pkg_resources import EntryPoint\n\nfrom datalad.api import Dataset\nfrom datalad.utils import (\n    chpwd,\n    swallow_logs,\n    swallow_outputs,\n)\nfrom datalad.tests.utils import (\n    assert_equal,\n    assert_in,\n    assert_re_in,\n    assert_is_generator,\n    assert_raises,\n    assert_repo_status,\n    assert_result_count,\n    eq_,\n    known_failure_githubci_win,\n    ok_file_under_git,\n    patch_config,\n    SkipTest,\n    with_tempfile,\n    with_testsui,\n)\nfrom datalad.support.exceptions import NoDatasetFound\n\nfrom datalad.api import search\n\nfrom datalad.metadata.search import (\n    _listdict2dictlist,\n    _meta2autofield_dict,\n)\n\n\n@with_testsui(interactive=False)\n@with_tempfile(mkdir=True)\ndef test_search_outside1_noninteractive_ui(tdir):\n    # we should raise an informative exception\n    with chpwd(tdir):\n        with assert_raises(NoDatasetFound) as cme:\n            list(search(\"bu\"))\n        assert_in('run interactively', str(cme.exception))\n\n\n@with_tempfile(mkdir=True)\n@with_tempfile(mkdir=True)\ndef test_search_outside1(tdir, newhome):\n    with chpwd(tdir):\n        # should fail since directory exists, but not a dataset\n        # should not even waste our response ;)\n        with patch_config({'datalad.locations.default-dataset': newhome}):\n            gen = search(\"bu\", return_type='generator')\n            assert_is_generator(gen)\n            assert_raises(NoDatasetFound, next, gen)\n\n        # and if we point to some non-existing dataset\n        with assert_raises(ValueError):\n            next(search(\"bu\", dataset=newhome))\n\n\n@with_testsui(responses='yes')\n@with_tempfile(mkdir=True)\n@with_tempfile()\ndef test_search_outside1_install_default_ds(tdir, default_dspath):\n    with chpwd(tdir):\n        # let's mock out even actual install/search calls\n        with \\\n            patch_config({'datalad.locations.default-dataset': default_dspath}), \\\n            patch('datalad.api.install',\n                  return_value=Dataset(default_dspath)) as mock_install, \\\n            patch('datalad.distribution.dataset.Dataset.search',\n                  new_callable=_mock_search):\n            _check_mocked_install(default_dspath, mock_install)\n\n            # now on subsequent run, we want to mock as if dataset already exists\n            # at central location and then do search again\n            from datalad.ui import ui\n            ui.add_responses('yes')\n            mock_install.reset_mock()\n            with patch(\n                    'datalad.distribution.dataset.Dataset.is_installed',\n                    True):\n                _check_mocked_install(default_dspath, mock_install)\n\n            # and what if we say \"no\" to install?\n            ui.add_responses('no')\n            mock_install.reset_mock()\n            with assert_raises(NoDatasetFound):\n                list(search(\".\"))\n\n            # and if path exists and is a valid dataset and we say \"no\"\n            Dataset(default_dspath).create()\n            ui.add_responses('no')\n            mock_install.reset_mock()\n            with assert_raises(NoDatasetFound):\n                list(search(\".\"))\n\n\n_mocked_search_results = [\n    {\n        'action': 'search',\n        'status': 'ok',\n        'path': 'ds1',  # this is wrong and must be an abspath\n        'matched': {'f': 'v'},  # this has nothing to do with the actual output\n    },\n    {\n        'action': 'search',\n        'status': 'ok',\n        'path': 'd2/ds2',  # this is wrong and must be an abspath\n        'matched': {'f1': 'v1'}  # this has nothing to do with the actual output,\n    },\n]\n\n\nclass _mock_search(object):\n    def __call__(*args, **kwargs):\n        for report in _mocked_search_results:\n            yield report\n\n\ndef _check_mocked_install(default_dspath, mock_install):\n    gen = search(\".\", return_type='generator')\n    assert_is_generator(gen)\n    # we no longer do any custom path tune up from the one returned by search\n    # so should match what search returns\n    assert_equal(\n        list(gen), [report\n                    for report in _mocked_search_results])\n    mock_install.assert_called_once_with(default_dspath, source='///')\n\n\n@with_tempfile\ndef test_search_non_dataset(tdir):\n    from datalad.support.gitrepo import GitRepo\n    GitRepo(tdir, create=True)\n    with assert_raises(NoDatasetFound) as cme:\n        list(search('smth', dataset=tdir))\n    # Should instruct user how that repo could become a datalad dataset\n    assert_in(\"datalad create --force\", str(cme.exception))\n\n\n@known_failure_githubci_win\n@with_tempfile(mkdir=True)\ndef test_within_ds_file_search(path):\n    try:\n        import mutagen\n    except ImportError:\n        raise SkipTest\n    ds = Dataset(path).create(force=True)\n    # override default and search for datasets and files for this test\n    for m in ('egrep', 'textblob', 'autofield'):\n        ds.config.add(\n            'datalad.search.index-{}-documenttype'.format(m), 'all',\n            where='dataset')\n    ds.config.add('datalad.metadata.nativetype', 'audio', where='dataset')\n    makedirs(opj(path, 'stim'))\n    for src, dst in (\n            ('audio.mp3', opj('stim', 'stim1.mp3')),):\n        copy(\n            opj(dirname(dirname(__file__)), 'tests', 'data', src),\n            opj(path, dst))\n    ds.save()\n    ok_file_under_git(path, opj('stim', 'stim1.mp3'), annexed=True)\n    # If it is not under annex, below addition of metadata silently does\n    # not do anything\n    ds.repo.set_metadata(\n        opj('stim', 'stim1.mp3'), init={'importance': 'very'})\n    ds.aggregate_metadata()\n    assert_repo_status(ds.path)\n    # basic sanity check on the metadata structure of the dataset\n    dsmeta = ds.metadata('.', reporton='datasets')[0]['metadata']\n    for src in ('audio',):\n        # something for each one\n        assert_in(src, dsmeta)\n        # each src declares its own context\n        assert_in('@context', dsmeta[src])\n        # we have a unique content metadata summary for each src\n        assert_in(src, dsmeta['datalad_unique_content_properties'])\n\n    # test default behavior\n    with swallow_outputs() as cmo:\n        ds.search(show_keys='name', mode='textblob')\n\n        assert_in(\"\"\"\\\nid\nmeta\nparentds\npath\ntype\n\"\"\", cmo.out)\n\n    target_out = \"\"\"\\\nannex.importance\nannex.key\naudio.bitrate\naudio.duration(s)\naudio.format\naudio.music-Genre\naudio.music-album\naudio.music-artist\naudio.music-channels\naudio.music-sample_rate\naudio.name\naudio.tracknumber\ndatalad_core.id\ndatalad_core.refcommit\nid\nparentds\npath\ntype\n\"\"\"\n\n    # test default behavior while limiting set of keys reported\n    with swallow_outputs() as cmo:\n        ds.search(['\\.id', 'artist$'], show_keys='short')\n        out_lines = [l for l in cmo.out.split(os.linesep) if l]\n        # test that only the ones matching were returned\n        assert_equal(\n            [l for l in out_lines if not l.startswith(' ')],\n            ['audio.music-artist', 'datalad_core.id']\n        )\n        # more specific test which would also test formatting\n        assert_equal(\n            out_lines,\n            ['audio.music-artist',\n             ' in  1 datasets', \" has 1 unique values: 'dlartist'\",\n             'datalad_core.id',\n             ' in  1 datasets',\n             # we have them sorted\n             \" has 1 unique values: '%s'\" % ds.id\n             ]\n        )\n\n    with assert_raises(ValueError) as cme:\n        ds.search('*wrong')\n    assert_re_in(\n        r\"regular expression '\\(\\?i\\)\\*wrong' \\(original: '\\*wrong'\\) is incorrect: \",\n        str(cme.exception))\n\n    # check generated autofield index keys\n    with swallow_outputs() as cmo:\n        ds.search(mode='autofield', show_keys='name')\n        # it is impossible to assess what is different from that dump\n        assert_in(target_out, cmo.out)\n\n    assert_result_count(ds.search('blablob#'), 0)\n    # now check that we can discover things from the aggregated metadata\n    for mode, query, hitpath, matched in (\n        ('egrep',\n         ':mp3',\n         opj('stim', 'stim1.mp3'),\n         {'audio.format': 'mp3'}),\n        # same as above, leading : is stripped, in indicates \"ALL FIELDS\"\n        ('egrep',\n         'mp3',\n         opj('stim', 'stim1.mp3'),\n         {'audio.format': 'mp3'}),\n        # same as above, but with AND condition\n        # get both matches\n        ('egrep',\n         ['mp3', 'type:file'],\n         opj('stim', 'stim1.mp3'),\n         {'type': 'file', 'audio.format': 'mp3'}),\n        # case insensitive search\n        ('egrep',\n         'mp3',\n         opj('stim', 'stim1.mp3'),\n         {'audio.format': 'mp3'}),\n        # field selection by expression\n        ('egrep',\n         'audio\\.+:mp3',\n         opj('stim', 'stim1.mp3'),\n         {'audio.format': 'mp3'}),\n        # random keyword query\n        ('textblob',\n         'mp3',\n         opj('stim', 'stim1.mp3'),\n         {'meta': 'mp3'}),\n        # report which field matched with auto-field\n        ('autofield',\n         'mp3',\n         opj('stim', 'stim1.mp3'),\n         {'audio.format': 'mp3'}),\n        # XXX next one is not supported by current text field analyser\n        # decomposes the mime type in [mime, audio, mp3]\n        # ('autofield',\n        # \"'mime:audio/mp3'\",\n        # opj('stim', 'stim1.mp3'),\n        # 'audio.format', 'mime:audio/mp3'),\n        # but this one works\n        ('autofield',\n         \"'mime audio mp3'\",\n         opj('stim', 'stim1.mp3'),\n         {'audio.format': 'mp3'}),\n        # TODO extend with more complex queries to test whoosh\n        # query language configuration\n    ):\n        res = ds.search(query, mode=mode, full_record=True)\n        assert_result_count(\n            res, 1, type='file', path=opj(ds.path, hitpath),\n            # each file must report the ID of the dataset it is from, critical for\n            # discovering related content\n            dsid=ds.id)\n        # in egrep we currently do not search unique values\n        # and the queries above aim at files\n        assert_result_count(res, 1 if mode == 'egrep' else 2)\n        if mode != 'egrep':\n            assert_result_count(\n                res, 1, type='dataset', path=ds.path, dsid=ds.id)\n        # test the key and specific value of the match\n        for matched_key, matched_val in matched.items():\n            assert_in(matched_key, res[-1]['query_matched'])\n            assert_equal(res[-1]['query_matched'][matched_key], matched_val)\n\n    # test a suggestion msg being logged if no hits and key is a bit off\n    with swallow_logs(new_level=logging.INFO) as cml:\n        res = ds.search('audio.formats:mp3 audio.bitsrate:1', mode='egrep')\n        assert not res\n        assert_in('Did you mean any of', cml.out)\n        assert_in('audio.format', cml.out)\n        assert_in('audio.bitrate', cml.out)\n\n\ndef test_listdict2dictlist():\n    f = _listdict2dictlist\n    l1 = [1, 3, [1, 'a']]\n    assert f(l1) is l1, \"we return it as is if no emb dict\"\n    eq_(f([{1: 2}]), {1: 2})  # inside out no need for a list\n    # inside out, join into the list, skip entry with a list, or space\n    eq_(f([{1: [2, 3], 'a': 1}, {'a': 2, 'c': ''}]), {'a': [1, 2]})\n\n\ndef test_meta2autofield_dict():\n    # Just a test that we would obtain the value stored for that extractor\n    # instead of what unique values it already had (whatever that means)\n    eq_(\n        _meta2autofield_dict({\n            'datalad_unique_content_properties':\n                {'extr1': {\"prop1\": \"v1\"}},\n            'extr1': {'prop1': 'value'}}),\n        {'extr1.prop1': 'value'}\n    )\n\n\ndef test_external_indexer():\n    \"\"\" check that external indexer are called \"\"\"\n    class MockedIndexer:\n        def __init__(self, metadata_format_name: str):\n            pass\n\n        def create_index(self, metadata):\n            yield from {\n                \"jubel\": 1,\n                \"trubel\": 2\n            }.items()\n\n    class MockedEntryPoint(EntryPoint):\n        def __init__(self):\n            pass\n\n        def load(self, *args):\n            return MockedIndexer\n\n    def _mocked_iter_entry_points(group, metadata):\n        yield MockedEntryPoint()\n\n    with patch('pkg_resources.iter_entry_points',\n               MagicMock(side_effect=_mocked_iter_entry_points)):\n        index = _meta2autofield_dict({\n            'datalad_unique_content_properties': {\n                'extr1': {\n                    \"prop1\": \"v1\"\n                }\n            },\n            'extr1': {\n                'prop1': 'value'\n            }\n        })\n\n    eq_(\n        index,\n        {\n            'extr1.jubel': '1',\n            'extr1.trubel': '2'\n        }\n    )\n\n\ndef test_faulty_external_indexer():\n    \"\"\" check that generic indexer is called on external indexer faults \"\"\"\n    class MockedEntryPoint(EntryPoint):\n        def __init__(self):\n            self.name = 'MockedEntryPoint'\n\n        def load(self, *args):\n            raise Exception('Mocked indexer error')\n\n    def _mocked_iter_entry_points(group, metadata):\n        yield MockedEntryPoint()\n\n    with patch('pkg_resources.iter_entry_points',\n               MagicMock(side_effect=_mocked_iter_entry_points)):\n        index = _meta2autofield_dict({\n            'datalad_unique_content_properties': {\n                'extr1': {\n                    \"prop1\": \"v1\"\n                }\n            },\n            'extr1': {\n                'prop1': 'value'\n            }\n        })\n\n    eq_(\n        index,\n        {\n            'extr1.prop1': 'value'\n        }\n    )\n",
        "import pytest\nimport tempfile\nimport os\n\n\n@pytest.fixture\ndef path():\n    with tempfile.TemporaryDirectory() as tmpdir:\n        yield tmpdir\n\n\n@pytest.fixture(autouse=True)\ndef setup_git_config():\n    with tempfile.TemporaryDirectory() as home:\n        old_home = os.environ.get(\"HOME\")\n        os.environ[\"HOME\"] = home\n        os.system('git config --global user.name \"DataLad Tester\"')\n        os.system('git config --global user.email \"test@example.com\"')\n        yield\n        if old_home:\n            os.environ[\"HOME\"] = old_home\n"
    ],
    "test_file_names": [
        "test_1.py",
        "conftest.py"
    ],
    "setup_res_code": 0,
    "setup_res_stdout": "Starting Datalad installation attempts...\nVerifying installation...\nDatalad version: 0.13.3.dev330\n✅ Installation successful!\nSuccessfully installed Datalad using Python 3.9\n",
    "setup_res_stderr": "Using CPython 3.9.21\nCreating virtual environment at: .venv\nActivate with: source .venv/bin/activate\nResolved 25 packages in 58ms\nInstalled 25 packages in 5.00s\n + coverage==7.6.10\n + docopt==0.6.2\n + exceptiongroup==1.2.2\n + grabbit==0.2.6\n + importlib-resources==6.4.5\n + iniconfig==2.0.0\n + nibabel==5.3.2\n + num2words==0.5.14\n + numpy==2.0.2\n + packaging==24.2\n + pandas==2.2.3\n + patsy==1.0.1\n + pluggy==1.5.0\n + pybids==0.6.5\n + pytest==8.3.4\n + pytest-cov==6.0.0\n + python-dateutil==2.9.0.post0\n + pytz==2024.2\n + scipy==1.13.1\n + setuptools==75.6.0\n + six==1.17.0\n + tomli==2.2.1\n + typing-extensions==4.12.2\n + tzdata==2024.2\n + zipp==3.21.0\nResolved 63 packages in 4.23s\nPrepared 1 package in 3.92s\nInstalled 57 packages in 781ms\n + annexremote==1.6.6\n + appdirs==1.4.4\n + backports-tarfile==1.2.0\n + beautifulsoup4==4.12.3\n + boto==2.49.0\n + certifi==2024.12.14\n + cffi==1.17.1\n + chardet==5.2.0\n + charset-normalizer==3.4.1\n + citeproc-py==0.6.0\n + cryptography==44.0.0\n + datalad==0.13.3 (from file:///home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_d1a45eaa510354812258cd022388cd202bff9080)\n + deprecated==1.2.15\n + distro==1.9.0\n + duecredit==0.10.2\n + exifread==3.0.0\n + fasteners==0.19\n + httpretty==1.1.4\n + humanize==4.11.0\n + idna==3.10\n + importlib-metadata==8.5.0\n + iso8601==2.1.0\n + jaraco-classes==3.4.0\n + jaraco-context==6.0.1\n + jaraco-functools==4.1.0\n + jeepney==0.8.0\n + jsmin==3.0.1\n + keyring==25.6.0\n + keyrings-alt==5.0.2\n + looseversion==1.3.0\n + lxml==5.3.0\n + more-itertools==10.5.0\n + msgpack==1.1.0\n + multidict==6.1.0\n + mutagen==1.47.0\n + nose==1.3.7\n + patool==1.15.0\n + pillow==11.1.0\n + propcache==0.2.1\n + pycparser==2.22\n + pygithub==2.5.0\n + pyjwt==2.10.1\n + pynacl==1.5.0\n + pyperclip==1.9.0\n + python-xmp-toolkit==2.0.2\n + pyyaml==6.0.2\n + requests==2.32.3\n + requests-ftp==0.3.1\n + secretstorage==3.3.3\n + simplejson==3.19.3\n + soupsieve==2.6\n + tqdm==4.67.1\n + urllib3==1.26.20\n + vcrpy==7.0.0\n + whoosh==2.7.4\n + wrapt==1.17.0\n + yarl==1.18.3\nResolved 97 packages in 241ms\nInstalled 32 packages in 941ms\n + alabaster==0.7.16\n + asv==0.6.4\n + asv-runner==0.2.1\n + babel==2.16.0\n + build==1.2.2.post1\n + distlib==0.3.9\n + docutils==0.21.2\n + filelock==3.16.1\n + gprof2dot==2024.6.6\n + imagesize==1.4.1\n + jinja2==3.1.5\n + json5==0.10.0\n + markupsafe==3.0.2\n + nose-timer==1.0.1\n + platformdirs==4.3.6\n + psutil==6.1.1\n + pygments==2.18.0\n + pympler==1.1\n + pypandoc==1.14\n + pyproject-hooks==1.2.0\n + snowballstemmer==2.2.0\n + sphinx==7.4.7\n + sphinx-rtd-theme==3.0.2\n + sphinxcontrib-applehelp==2.0.0\n + sphinxcontrib-devhelp==2.0.0\n + sphinxcontrib-htmlhelp==2.1.0\n + sphinxcontrib-jquery==4.1\n + sphinxcontrib-jsmath==1.0.1\n + sphinxcontrib-qthelp==2.0.0\n + sphinxcontrib-serializinghtml==2.0.0\n + tabulate==0.9.0\n + virtualenv==20.28.0\n",
    "new_commit_res_code": 1,
    "new_commit_res_stdout": "============================= test session starts ==============================\nplatform linux -- Python 3.9.21, pytest-8.3.4, pluggy-1.5.0\nrootdir: /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_d1a45eaa510354812258cd022388cd202bff9080\nconfigfile: pyproject.toml\nplugins: cov-6.0.0\ncollected 9 items\n\nr2e_tests/test_1.py EEEEF....                                            [100%]\n\n==================================== ERRORS ====================================\n___________ ERROR at setup of test_search_outside1_noninteractive_ui ___________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_d1a45eaa510354812258cd022388cd202bff9080/r2e_tests/test_1.py, line 56\n  @with_testsui(interactive=False)\n  @with_tempfile(mkdir=True)\n  def test_search_outside1_noninteractive_ui(tdir):\nE       fixture 'tdir' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_d1a45eaa510354812258cd022388cd202bff9080/r2e_tests/test_1.py:56\n____________________ ERROR at setup of test_search_outside1 ____________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_d1a45eaa510354812258cd022388cd202bff9080/r2e_tests/test_1.py, line 66\n  @with_tempfile(mkdir=True)\n  @with_tempfile(mkdir=True)\n  def test_search_outside1(tdir, newhome):\nE       fixture 'tdir' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_d1a45eaa510354812258cd022388cd202bff9080/r2e_tests/test_1.py:66\n__________ ERROR at setup of test_search_outside1_install_default_ds ___________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_d1a45eaa510354812258cd022388cd202bff9080/r2e_tests/test_1.py, line 82\n  @with_testsui(responses='yes')\n  @with_tempfile(mkdir=True)\n  @with_tempfile()\n  def test_search_outside1_install_default_ds(tdir, default_dspath):\nE       fixture 'tdir' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_d1a45eaa510354812258cd022388cd202bff9080/r2e_tests/test_1.py:82\n__________________ ERROR at setup of test_search_non_dataset ___________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_d1a45eaa510354812258cd022388cd202bff9080/r2e_tests/test_1.py, line 153\n  @with_tempfile\n  def test_search_non_dataset(tdir):\nE       fixture 'tdir' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_d1a45eaa510354812258cd022388cd202bff9080/r2e_tests/test_1.py:153\n=================================== FAILURES ===================================\n__________________________ test_within_ds_file_search __________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annexdwigevm6'}\nfilename = '/tmp/datalad_temp_test_within_ds_file_searchho8u0os1'\n\n    @wraps(t)\n    def  _wrap_with_tempfile(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_within_ds_file_search() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:731: TypeError\n==================================== PASSES ====================================\n_________________________ test_faulty_external_indexer _________________________\n----------------------------- Captured stderr call -----------------------------\n[WARNING] Failed to load indexer MockedEntryPoint: Mocked indexer error [test_1.py:load:417] \n------------------------------ Captured log call -------------------------------\nWARNING  datalad.metadata.search:search.py:187 Failed to load indexer MockedEntryPoint: Mocked indexer error [test_1.py:load:417]\n=========================== short test summary info ============================\nPASSED r2e_tests/test_1.py::test_listdict2dictlist\nPASSED r2e_tests/test_1.py::test_meta2autofield_dict\nPASSED r2e_tests/test_1.py::test_external_indexer\nPASSED r2e_tests/test_1.py::test_faulty_external_indexer\nERROR r2e_tests/test_1.py::test_search_outside1_noninteractive_ui\nERROR r2e_tests/test_1.py::test_search_outside1\nERROR r2e_tests/test_1.py::test_search_outside1_install_default_ds\nERROR r2e_tests/test_1.py::test_search_non_dataset\nFAILED r2e_tests/test_1.py::test_within_ds_file_search - TypeError: test_with...\n==================== 1 failed, 4 passed, 4 errors in 2.43s =====================\n",
    "new_commit_res_stderr": "",
    "old_commit_res_code": 1,
    "old_commit_res_stdout": "============================= test session starts ==============================\nplatform linux -- Python 3.9.21, pytest-8.3.4, pluggy-1.5.0\nrootdir: /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_d1a45eaa510354812258cd022388cd202bff9080\nconfigfile: pyproject.toml\nplugins: cov-6.0.0\ncollected 9 items\n\nr2e_tests/test_1.py EEEEF..F.                                            [100%]\n\n==================================== ERRORS ====================================\n___________ ERROR at setup of test_search_outside1_noninteractive_ui ___________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_d1a45eaa510354812258cd022388cd202bff9080/r2e_tests/test_1.py, line 56\n  @with_testsui(interactive=False)\n  @with_tempfile(mkdir=True)\n  def test_search_outside1_noninteractive_ui(tdir):\nE       fixture 'tdir' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_d1a45eaa510354812258cd022388cd202bff9080/r2e_tests/test_1.py:56\n____________________ ERROR at setup of test_search_outside1 ____________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_d1a45eaa510354812258cd022388cd202bff9080/r2e_tests/test_1.py, line 66\n  @with_tempfile(mkdir=True)\n  @with_tempfile(mkdir=True)\n  def test_search_outside1(tdir, newhome):\nE       fixture 'tdir' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_d1a45eaa510354812258cd022388cd202bff9080/r2e_tests/test_1.py:66\n__________ ERROR at setup of test_search_outside1_install_default_ds ___________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_d1a45eaa510354812258cd022388cd202bff9080/r2e_tests/test_1.py, line 82\n  @with_testsui(responses='yes')\n  @with_tempfile(mkdir=True)\n  @with_tempfile()\n  def test_search_outside1_install_default_ds(tdir, default_dspath):\nE       fixture 'tdir' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_d1a45eaa510354812258cd022388cd202bff9080/r2e_tests/test_1.py:82\n__________________ ERROR at setup of test_search_non_dataset ___________________\nfile /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_d1a45eaa510354812258cd022388cd202bff9080/r2e_tests/test_1.py, line 153\n  @with_tempfile\n  def test_search_non_dataset(tdir):\nE       fixture 'tdir' not found\n>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, path, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, setup_git_config, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory\n>       use 'pytest --fixtures [testpath]' for help on them.\n\n/home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_d1a45eaa510354812258cd022388cd202bff9080/r2e_tests/test_1.py:153\n=================================== FAILURES ===================================\n__________________________ test_within_ds_file_search __________________________\n\narg = (), kw = {'path': '/tmp/tmp-page2annexu0rbyr01'}\nfilename = '/tmp/datalad_temp_test_within_ds_file_search6er22hsi'\n\n    @wraps(t)\n    def  _wrap_with_tempfile(*arg, **kw):\n        if 'dir' not in tkwargs.keys():\n            # if not specified otherwise, respect datalad.tests.temp.dir config\n            # as this is a test helper\n            tkwargs['dir'] = dl_cfg.get(\"datalad.tests.temp.dir\")\n        with make_tempfile(wrapped=t, **tkwargs) as filename:\n>           return t(*(arg + (filename,)), **kw)\nE           TypeError: test_within_ds_file_search() got multiple values for argument 'path'\n\ndatalad/tests/utils.py:731: TypeError\n____________________________ test_external_indexer _____________________________\n\n    def test_external_indexer():\n        \"\"\" check that external indexer are called \"\"\"\n        class MockedIndexer:\n            def __init__(self, metadata_format_name: str):\n                pass\n    \n            def create_index(self, metadata):\n                yield from {\n                    \"jubel\": 1,\n                    \"trubel\": 2\n                }.items()\n    \n        class MockedEntryPoint(EntryPoint):\n            def __init__(self):\n                pass\n    \n            def load(self, *args):\n                return MockedIndexer\n    \n        def _mocked_iter_entry_points(group, metadata):\n            yield MockedEntryPoint()\n    \n        with patch('pkg_resources.iter_entry_points',\n                   MagicMock(side_effect=_mocked_iter_entry_points)):\n            index = _meta2autofield_dict({\n                'datalad_unique_content_properties': {\n                    'extr1': {\n                        \"prop1\": \"v1\"\n                    }\n                },\n                'extr1': {\n                    'prop1': 'value'\n                }\n            })\n    \n>       eq_(\n            index,\n            {\n                'extr1.jubel': '1',\n                'extr1.trubel': '2'\n            }\n        )\n\nr2e_tests/test_1.py:401: \n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \n\na = {'extr1.prop1': 'value'}, b = {'extr1.jubel': '1', 'extr1.trubel': '2'}\nmsg = None\n\n    def eq_(a, b, msg=None):\n        \"\"\"Shorthand for 'assert a == b, \"%r != %r\" % (a, b)\n        \"\"\"\n        if not a == b:\n>           raise AssertionError(msg or \"%r != %r\" % (a, b))\nE           AssertionError: {'extr1.prop1': 'value'} != {'extr1.jubel': '1', 'extr1.trubel': '2'}\n\n.venv/lib/python3.9/site-packages/nose/tools/trivial.py:29: AssertionError\n==================================== PASSES ====================================\n=========================== short test summary info ============================\nPASSED r2e_tests/test_1.py::test_listdict2dictlist\nPASSED r2e_tests/test_1.py::test_meta2autofield_dict\nPASSED r2e_tests/test_1.py::test_faulty_external_indexer\nERROR r2e_tests/test_1.py::test_search_outside1_noninteractive_ui\nERROR r2e_tests/test_1.py::test_search_outside1\nERROR r2e_tests/test_1.py::test_search_outside1_install_default_ds\nERROR r2e_tests/test_1.py::test_search_non_dataset\nFAILED r2e_tests/test_1.py::test_within_ds_file_search - TypeError: test_with...\nFAILED r2e_tests/test_1.py::test_external_indexer - AssertionError: {'extr1.p...\n==================== 2 failed, 3 passed, 4 errors in 1.18s =====================\n",
    "old_commit_res_stderr": ""
}
datalad/metadata/indexers/__init__.py datalad/metadata/indexers/base.py datalad/metadata/search.py datalad/metadata/tests/test_search.py
We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
[ISSUE]

**Title:** External Metadata Indexers Not Invoked, Causing Incomplete Metadata

**Description:**
When using external metadata indexers, the additional metadata they are supposed to provide is missing. Instead of including the key-value pairs from the external indexer, only the original metadata is present.

**Example:**
```python
metadata = {
    'datalad_unique_content_properties': {
        'extr1': {'prop1': 'v1'}
    },
    'extr1': {'prop1': 'value'}
}

processed_metadata = _meta2autofield_dict(metadata)

print(processed_metadata)
# Output: {'extr1.prop1': 'value'}
```

**Expected Behavior:**
The external indexer should process the `extr1` metadata and add the `jubel` and `trubel` fields, resulting in:
```python
{'extr1.jubel': '1', 'extr1.trubel': '2'}
```

**Actual Behavior:**
Only the original `extr1.prop1` key is present in the processed metadata, and the fields from the external indexer are missing.

[/ISSUE]
--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.

--- BEGIN FILE ---
```
### datalad/metadata/search.py
# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-
# ex: set sts=4 ts=4 sw=4 noet:
# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
#
#   See COPYING file distributed along with the datalad package for the
#   copyright and license terms.
#
# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
"""Interface for managing metadata
"""

__docformat__ = 'restructuredtext'

import collections
import logging
from datalad.log import log_progress
lgr = logging.getLogger('datalad.metadata.search')

import os
import re
from functools import partial
from os.path import join as opj, exists
from os.path import relpath
from os.path import normpath
import sys
from time import time

from pkg_resources import EntryPoint, iter_entry_points

from datalad import cfg
from datalad.interface.base import Interface
from datalad.interface.base import build_doc
from datalad.interface.utils import eval_results
from datalad.distribution.dataset import Dataset
from datalad.distribution.dataset import datasetmethod, EnsureDataset, \
    require_dataset
from datalad.support.gitrepo import GitRepo
from datalad.support.param import Parameter
from datalad.support.constraints import EnsureNone
from datalad.support.constraints import EnsureInt

from datalad.consts import SEARCH_INDEX_DOTGITDIR
from datalad.utils import (
    as_unicode,
    ensure_list,
    ensure_unicode,
    get_suggestions_msg,
    shortened_repr,
    unicode_srctypes,
)
from datalad.support.exceptions import NoDatasetFound
from datalad.ui import ui
from datalad.dochelpers import single_or_plural
from datalad.dochelpers import exc_str
from datalad.metadata.metadata import query_aggregated_metadata

# TODO: consider using plain as_unicode, without restricting
# the types?
_any2unicode = partial(as_unicode, cast_types=(int, float, tuple, list, dict))


def _listdict2dictlist(lst, strict=True):
    """Helper to deal with DataLad's unique value reports

    Parameters
    ----------
    lst : list
      List of dicts
    strict : bool
      In strict mode any dictionary items that doesn't have a simple value
      (but another container) is discarded. In non-strict mode, arbitrary
      levels of nesting are preserved, and no unique-ification of values
      is performed. The latter can be used when it is mostly (or merely)
      interesting to get a list of metadata keys.
    """
    # unique values that we got, always a list
    if all(not isinstance(uval, dict) for uval in lst):
        # no nested structures, take as is
        return lst

    type_excluder = (tuple, list)
    if strict:
        type_excluder += (dict,)
    # we need to turn them inside out, instead of a list of
    # dicts, we want a dict where values are lists, because we
    # cannot handle hierarchies in a tabular search index
    # if you came here to find that out, go ahead and use a graph
    # DB/search
    udict = {}
    for uv in lst:
        if not isinstance(uv, dict):
            # we cannot mix and match, discard any scalar values
            # in favor of the structured metadata
            continue
        for k, v in uv.items():
            if isinstance(v, type_excluder):
                # this is where we draw the line, two levels of
                # nesting. whoosh can only handle string values
                # injecting a stringified blob of something doesn't
                # really enable anything useful -> graph search
                continue
            if v == "":
                # no cruft
                continue
            if strict:
                # no duplicate values, only hashable stuff
                uvals = udict.get(k, set())
                uvals.add(v)
            else:
                # whatever it is, we'll take it
                uvals = udict.get(k, [])
                uvals.append(v)
            udict[k] = uvals
    return {
        # set not good for JSON, have plain list
        k: list(v) if len(v) > 1 else list(v)[0]
        for k, v in udict.items()
        # do not accumulate cruft
        if len(v)}


def _meta2autofield_dict(meta, val2str=True, schema=None, consider_ucn=True):
    """Takes care of dtype conversion into unicode, potential key mappings
    and concatenation of sequence-type fields into CSV strings

    - if `consider_ucn` (default) it would copy keys from
      datalad_unique_content_properties into `meta` for that extractor
    - ... TODO ...
    """
    if consider_ucn:
        # loop over all metadata sources and the report of their unique values
        ucnprops = meta.get("datalad_unique_content_properties", {})
        for src, umeta in ucnprops.items():
            srcmeta = meta.get(src, {})
            for uk in umeta:
                if uk in srcmeta:
                    # we have a real entry for this key in the dataset metadata
                    # ignore any generated unique value list in favor of the
                    # tailored data
                    continue
                srcmeta[uk] = _listdict2dictlist(umeta[uk], strict=False) if umeta[uk] is not None else None
            if src not in meta and srcmeta:
                meta[src] = srcmeta  # assign the new one back

    srcmeta = None   # for paranoids to avoid some kind of manipulation of the last

    def _deep_kv(basekey, dct):
        """Return key/value pairs of any depth following a rule for key
        composition

        dct must be a dict
        """
        for k, v in dct.items():
            if (k != '@id' and k.startswith('@')) or k == 'datalad_unique_content_properties':
                # ignore all JSON-LD specials, but @id
                continue
            # TODO `k` might need remapping, if another key was already found
            # with the same definition
            key = u'{}{}'.format(
                basekey,
                # replace now special chars, and avoid spaces
                # `os.sep` needs to go, because whoosh uses the field name for
                # temp files during index staging, and trips over absent "directories"
                # TODO maybe even kill parentheses
                # TODO actually, it might be better to have an explicit whitelist
                k.lstrip('@').replace(os.sep, '_').replace(' ', '_').replace('-', '_').replace('.', '_').replace(':', '-')
            )
            if isinstance(v, list):
                v = _listdict2dictlist(v)

            if isinstance(v, dict):
                # dive
                for i in _deep_kv('{}.'.format(key), v):
                    yield i
            else:
                yield key, v

    def get_indexer_for(metadata_format_name: str) -> callable:
        indexer = (tuple(iter_entry_points('datalad.metadata.indexers', metadata_format_name)) or (None,))[0]
        if isinstance(indexer, EntryPoint):
            try:
                indexer_object = indexer.load()(metadata_format_name)
                return indexer_object.create_index
            except Exception as e:
                lgr.warning('Failed to load indexer %s: %s', indexer.name, exc_str(e))
        return lambda metadata: _deep_kv('', metadata)

    def _iterable_to_unicode_string(value, perform_operation):
        if perform_operation:
            if isinstance(value, (list, tuple)):
                return u' '.join(_any2unicode(i) for i in value)
            return _any2unicode(value)
        return value

    meta = meta or {}
    return {
        # Collect all meta-items which have a non-dict value type and where
        # the key not absent in a given schema.
        **{
            key: _iterable_to_unicode_string(value, val2str)
            for key, value in filter(lambda kv: not isinstance(kv[1], dict), meta.items())
            if schema is None or key in schema
        },

        # Collect all meta-items which have a dict value type and where
        # the key is neither 'datalad_unique_content_properties' nor absent
        # in a given schema.
        # These values are considered as metadata, the keys are considered to
        # be the name of the extractor, i.e. the metadata_format_name, that created
        # the metadata.
        **{
            metadata_format_name + '.' + sub_key: _iterable_to_unicode_string(sub_key_value, val2str)
            for metadata_format_name, metadata_content in filter(
                lambda kv: isinstance(kv[1], dict) and kv[0] != 'datalad_unique_content_properties',
                meta.items()
            )
            for sub_key, sub_key_value in get_indexer_for(metadata_format_name)(metadata_content)
            if schema is None or metadata_format_name + '.' + sub_key in schema
        }
    }


def _search_from_virgin_install(dataset, query):
    #
    # this is to be nice to newbies
    #
    exc_info = sys.exc_info()
    if dataset is None:
        if not ui.is_interactive:
            raise NoDatasetFound(
                "No DataLad dataset found. Specify a dataset to be "
                "searched, or run interactively to get assistance "
                "installing a queriable superdataset."
            )
        # none was provided so we could ask user whether he possibly wants
        # to install our beautiful mega-duper-super-dataset?
        # TODO: following logic could possibly benefit other actions.
        DEFAULT_DATASET_PATH = cfg.obtain('datalad.locations.default-dataset')
        if os.path.exists(DEFAULT_DATASET_PATH):
            default_ds = Dataset(DEFAULT_DATASET_PATH)
            if default_ds.is_installed():
                if ui.yesno(
                    title="No DataLad dataset found at current location",
                    text="Would you like to search the DataLad "
                         "superdataset at %r?"
                          % DEFAULT_DATASET_PATH):
                    pass
                else:
                    raise exc_info[1]
            else:
                raise NoDatasetFound(
                    "No DataLad dataset found at current location. "
                    "The DataLad superdataset location %r exists, "
                    "but does not contain an dataset."
                    % DEFAULT_DATASET_PATH)
        elif ui.yesno(
                title="No DataLad dataset found at current location",
                text="Would you like to install the DataLad "
                     "superdataset at %r?"
                     % DEFAULT_DATASET_PATH):
            from datalad.api import install
            default_ds = install(DEFAULT_DATASET_PATH, source='///')
            ui.message(
                "From now on you can refer to this dataset using the "
                "label '///'"
            )
        else:
            raise exc_info[1]

        lgr.info(
            "Performing search using DataLad superdataset %r",
            default_ds.path
        )
        for res in default_ds.search(query):
            yield res
        return
    else:
        raise  # this function is called within exception handling block


class _Search(object):
    def __init__(self, ds, **kwargs):
        self.ds = ds
        self.documenttype = self.ds.config.obtain(
            'datalad.search.index-{}-documenttype'.format(self._mode_label),
            default=self._default_documenttype)

    def __call__(self, query, max_nresults=None):
        raise NotImplementedError

    @classmethod
    def _key_matches(cls, k, regexes):
        """Return which regex the key matches
        """
        for regex in regexes:
            if re.search(regex, k):
                return regex

    def show_keys(self, *args, **kwargs):
        raise NotImplementedError(args)

    def get_query(self, query):
        """Prepare query structure specific for a search backend.

        It can also memorize within instance some parameters of the last query
        which could be used to assist output formatting/structuring later on
        """
        raise NotImplementedError

    def get_nohits_msg(self):
        """Given what it knows, provide recommendation in the case of no hits"""
        return "No search hits, wrong query? " \
               "See 'datalad search --show-keys name' for known keys " \
               "and 'datalad search --help' on how to prepare your query."


class _WhooshSearch(_Search):
    def __init__(self, ds, force_reindex=False, **kwargs):
        super(_WhooshSearch, self).__init__(ds, **kwargs)

        self.idx_obj = None
        # where does the bunny have the eggs?

        self.index_dir = opj(str(self.ds.repo.dot_git), SEARCH_INDEX_DOTGITDIR)
        self._mk_search_index(force_reindex)

    def show_keys(self, mode, regexes=None):
        """

        Parameters
        ----------
        mode: {"name"}
        regexes: list of regex
          Which keys to bother working on
        """
        if mode != 'name':
            raise NotImplementedError(
                "ATM %s can show only names, so please use show_keys with 'name'"
                % self.__class__.__name__
            )
        for k in self.idx_obj.schema.names():
            if regexes and not self._key_matches(k, regexes):
                continue
            print(u'{}'.format(k))

    def get_query(self, query):
        # parse the query string
        self._mk_parser()
        # for convenience we accept any number of args-words from the
        # shell and put them together to a single string here
        querystr = ' '.join(ensure_list(query))
        # this gives a formal whoosh query
        wquery = self.parser.parse(querystr)
        return wquery

    def _meta2doc(self, meta, val2str=True, schema=None):
        raise NotImplementedError

    def _mk_schema(self):
        raise NotImplementedError

    def _mk_parser(self):
        raise NotImplementedError

    def _mk_search_index(self, force_reindex):
        """Generic entrypoint to index generation

        The actual work that determines the structure and content of the index
        is done by functions that are passed in as arguments

        `meta2doc` - must return dict for index document from result input
        """
        from whoosh import index as widx
        from .metadata import get_ds_aggregate_db_locations
        dbloc, db_base_path = get_ds_aggregate_db_locations(self.ds)
        # what is the lastest state of aggregated metadata
        metadata_state = self.ds.repo.get_last_commit_hexsha(relpath(dbloc, start=self.ds.path))
        # use location common to all index types, they would all invalidate
        # simultaneously
        stamp_fname = opj(self.index_dir, 'datalad_metadata_state')
        index_dir = opj(self.index_dir, self._mode_label)

        if (not force_reindex) and \
                exists(index_dir) and \
                exists(stamp_fname) and \
                open(stamp_fname).read() == metadata_state:
            try:
                # TODO check that the index schema is the same
                # as the one we would have used for reindexing
                # TODO support incremental re-indexing, whoosh can do it
                idx = widx.open_dir(index_dir)
                lgr.debug(
                    'Search index contains %i documents',
                    idx.doc_count())
                self.idx_obj = idx
                return
            except widx.LockError as e:
                raise e
            except widx.IndexError as e:
                # Generic index error.
                # we try to regenerate
                lgr.warning(
                    "Cannot open existing index %s (%s), will regenerate",
                    index_dir, exc_str(e)
                )
            except widx.IndexVersionError as e:  # (msg, version, release=None)
                # Raised when you try to open an index using a format that the
                # current version of Whoosh cannot read. That is, when the index
                # you're trying to open is either not backward or forward
                # compatible with this version of Whoosh.
                # we try to regenerate
                lgr.warning(exc_str(e))
                pass
            except widx.OutOfDateError as e:
                # Raised when you try to commit changes to an index which is not
                # the latest generation.
                # this should not happen here, but if it does ... KABOOM
                raise
            except widx.EmptyIndexError as e:
                # Raised when you try to work with an index that has no indexed
                # terms.
                # we can just continue with generating an index
                pass
            except ValueError as e:
                if 'unsupported pickle protocol' in str(e):
                    lgr.warning(
                        "Cannot open existing index %s (%s), will regenerate",
                        index_dir, exc_str(e)
                    )
                else:
                    raise

        lgr.info('{} search index'.format(
            'Rebuilding' if exists(index_dir) else 'Building'))

        if not exists(index_dir):
            os.makedirs(index_dir)

        # this is a pretty cheap call that just pull this info from a file
        dsinfo = self.ds.metadata(
            get_aggregates=True,
            return_type='list',
            result_renderer='disabled')

        self._mk_schema(dsinfo)

        idx_obj = widx.create_in(index_dir, self.schema)
        idx = idx_obj.writer(
            # cache size per process
            limitmb=cfg.obtain('datalad.search.indexercachesize'),
            # disable parallel indexing for now till #1927 is resolved
            ## number of processes for indexing
            #procs=multiprocessing.cpu_count(),
            ## write separate index segments in each process for speed
            ## asks for writer.commit(optimize=True)
            #multisegment=True,
        )

        # load metadata of the base dataset and what it knows about all its subdatasets
        # (recursively)
        old_idx_size = 0
        old_ds_rpath = ''
        idx_size = 0
        log_progress(
            lgr.info,
            'autofieldidxbuild',
            'Start building search index',
            total=len(dsinfo),
            label='Building search index',
            unit=' Datasets',
        )
        for res in query_aggregated_metadata(
                reporton=self.documenttype,
                ds=self.ds,
                aps=[dict(path=self.ds.path, type='dataset')],
                # MIH: I cannot see a case when we would not want recursion (within
                # the metadata)
                recursive=True):
            # this assumes that files are reported after each dataset report,
            # and after a subsequent dataset report no files for the previous
            # dataset will be reported again
            meta = res.get('metadata', {})
            doc = self._meta2doc(meta)
            admin = {
                'type': res['type'],
                'path': relpath(res['path'], start=self.ds.path),
            }
            if 'parentds' in res:
                admin['parentds'] = relpath(res['parentds'], start=self.ds.path)
            if admin['type'] == 'dataset':
                if old_ds_rpath:
                    lgr.debug(
                        'Added %s on dataset %s',
                        single_or_plural(
                            'document',
                            'documents',
                            idx_size - old_idx_size,
                            include_count=True),
                        old_ds_rpath)
                log_progress(lgr.info, 'autofieldidxbuild',
                             'Indexed dataset at %s', old_ds_rpath,
                             update=1, increment=True)
                old_idx_size = idx_size
                old_ds_rpath = admin['path']
                admin['id'] = res.get('dsid', None)

            doc.update({k: ensure_unicode(v) for k, v in admin.items()})
            lgr.debug("Adding document to search index: {}".format(doc))
            # inject into index
            idx.add_document(**doc)
            idx_size += 1

        if old_ds_rpath:
            lgr.debug(
                'Added %s on dataset %s',
                single_or_plural(
                    'document',
                    'documents',
                    idx_size - old_idx_size,
                    include_count=True),
                old_ds_rpath)

        lgr.debug("Committing index")
        idx.commit(optimize=True)
        log_progress(
            lgr.info, 'autofieldidxbuild', 'Done building search index')

        # "timestamp" the search index to allow for automatic invalidation
        with open(stamp_fname, 'w') as f:
            f.write(metadata_state)

        lgr.info('Search index contains %i documents', idx_size)
        self.idx_obj = idx_obj

    def __call__(self, query, max_nresults=None, force_reindex=False, full_record=False):
        if max_nresults is None:
            # mode default
            max_nresults = 20

        with self.idx_obj.searcher() as searcher:
            wquery = self.get_query(query)

            # perform the actual search
            hits = searcher.search(
                wquery,
                terms=True,
                limit=max_nresults if max_nresults > 0 else None)
            # report query stats
            topstr = '{} top {}'.format(
                max_nresults,
                single_or_plural('match', 'matches', max_nresults)
            )
            lgr.info('Query completed in {} sec.{}'.format(
                hits.runtime,
                ' Reporting {}.'.format(
                    ('up to ' + topstr)
                    if max_nresults > 0
                    else 'all matches'
                )
                if not hits.is_empty()
                else ' No matches.'
            ))

            if not hits:
                return

            nhits = 0
            annotated_hits = []
            # annotate hits for full metadata report
            for i, hit in enumerate(hits):
                annotated_hit = dict(
                    path=normpath(opj(self.ds.path, hit['path'])),
                    query_matched={ensure_unicode(k): ensure_unicode(v)
                                   if isinstance(v, unicode_srctypes) else v
                                   for k, v in hit.matched_terms()},
                    parentds=normpath(
                        opj(self.ds.path, hit['parentds'])) if 'parentds' in hit else None,
                    type=hit.get('type', None))
                if not full_record:
                    annotated_hit.update(
                        refds=self.ds.path,
                        action='search',
                        status='ok',
                        logger=lgr,
                    )
                    yield annotated_hit
                else:
                    annotated_hits.append(annotated_hit)
                nhits += 1
            if full_record:
                for res in query_aggregated_metadata(
                        # type is taken from hit record
                        reporton=None,
                        ds=self.ds,
                        aps=annotated_hits,
                        # never recursive, we have direct hits already
                        recursive=False):
                    res.update(
                        refds=self.ds.path,
                        action='search',
                        status='ok',
                        logger=lgr,
                    )
                    yield res

            if max_nresults and nhits == max_nresults:
                lgr.info(
                    "Reached the limit of {}, there could be more which "
                    "were not reported.".format(topstr)
                )


class _BlobSearch(_WhooshSearch):
    _mode_label = 'textblob'
    _default_documenttype = 'datasets'

    def _meta2doc(self, meta):
        # coerce the entire flattened metadata dict into a comma-separated string
        # that also includes the keys
        return dict(meta=u', '.join(
            u'{}: {}'.format(k, v)
            for k, v in _meta2autofield_dict(
                meta,
                val2str=True,
                schema=None).items()))

    def _mk_schema(self, dsinfo):
        from whoosh import fields as wf
        from whoosh.analysis import StandardAnalyzer

        # TODO support some customizable mapping to homogenize some metadata fields
        # onto a given set of index keys
        self.schema = wf.Schema(
            id=wf.ID,
            path=wf.ID(stored=True),
            type=wf.ID(stored=True),
            parentds=wf.ID(stored=True),
            meta=wf.TEXT(
                stored=False,
                analyzer=StandardAnalyzer(minsize=2))
        )

    def _mk_parser(self):
        from whoosh import qparser as qparse

        # use whoosh default query parser for now
        parser = qparse.QueryParser(
            "meta",
            schema=self.idx_obj.schema
        )
        parser.add_plugin(qparse.FuzzyTermPlugin())
        parser.remove_plugin_class(qparse.PhrasePlugin)
        parser.add_plugin(qparse.SequencePlugin())
        self.parser = parser


class _AutofieldSearch(_WhooshSearch):
    _mode_label = 'autofield'
    _default_documenttype = 'all'

    def _meta2doc(self, meta):
        return _meta2autofield_dict(meta, val2str=True, schema=self.schema)

    def _mk_schema(self, dsinfo):
        from whoosh import fields as wf
        from whoosh.analysis import SimpleAnalyzer

        # haven for terms that have been found to be undefined
        # (for faster decision-making upon next encounter)
        # this will harvest all discovered term definitions
        definitions = {
            '@id': 'unique identifier of an entity',
            # TODO make proper JSON-LD definition
            'path': 'path name of an entity relative to the searched base dataset',
            # TODO make proper JSON-LD definition
            'parentds': 'path of the datasets that contains an entity',
            # 'type' will not come from a metadata field, hence will not be detected
            'type': 'type of a record',
        }

        schema_fields = {
            n.lstrip('@'): wf.ID(stored=True, unique=n == '@id')
            for n in definitions}

        lgr.debug('Scanning for metadata keys')
        # quick 1st pass over all dataset to gather the needed schema fields
        log_progress(
            lgr.info,
            'idxschemabuild',
            'Start building search schema',
            total=len(dsinfo),
            label='Building search schema',
            unit=' Datasets',
        )
        for res in query_aggregated_metadata(
                # XXX TODO After #2156 datasets may not necessarily carry all
                # keys in the "unique" summary
                reporton='datasets',
                ds=self.ds,
                aps=[dict(path=self.ds.path, type='dataset')],
                recursive=True):
            meta = res.get('metadata', {})
            # no stringification of values for speed, we do not need/use the
            # actual values at this point, only the keys
            idxd = _meta2autofield_dict(meta, val2str=False)

            for k in idxd:
                schema_fields[k] = wf.TEXT(stored=False,
                                           analyzer=SimpleAnalyzer())
            log_progress(lgr.info, 'idxschemabuild',
                         'Scanned dataset at %s', res['path'],
                         update=1, increment=True)
        log_progress(
            lgr.info, 'idxschemabuild', 'Done building search schema')

        self.schema = wf.Schema(**schema_fields)

    def _mk_parser(self):
        from whoosh import qparser as qparse

        parser = qparse.MultifieldParser(
            self.idx_obj.schema.names(),
            self.idx_obj.schema)
        # XXX: plugin is broken in Debian's whoosh 2.7.0-2, but already fixed
        # upstream
        parser.add_plugin(qparse.FuzzyTermPlugin())
        parser.add_plugin(qparse.GtLtPlugin())
        parser.add_plugin(qparse.SingleQuotePlugin())
        # replace field defintion to allow for colons to be part of a field's name:
        parser.replace_plugin(qparse.FieldsPlugin(expr=r"(?P<text>[()<>.\w]+|[*]):"))
        self.parser = parser


class _EGrepCSSearch(_Search):
    _mode_label = 'egrepcs'
    _default_documenttype = 'datasets'

    def __init__(self, ds, **kwargs):
        super(_EGrepCSSearch, self).__init__(ds, **kwargs)
        self._queried_keys = None  # to be memoized by get_query

    # If there were custom "per-search engine" options, we could expose
    # --consider_ucn - search through unique content properties of the dataset
    #    which might be more computationally demanding
    def __call__(self, query, max_nresults=None, consider_ucn=False, full_record=True):
        if max_nresults is None:
            # no limit by default
            max_nresults = 0
        query = self.get_query(query)

        nhits = 0
        for res in query_aggregated_metadata(
                reporton=self.documenttype,
                ds=self.ds,
                aps=[dict(path=self.ds.path, type='dataset')],
                # MIH: I cannot see a case when we would not want recursion (within
                # the metadata)
                recursive=True):
            # this assumes that files are reported after each dataset report,
            # and after a subsequent dataset report no files for the previous
            # dataset will be reported again
            meta = res.get('metadata', {})
            # produce a flattened metadata dict to search through
            doc = _meta2autofield_dict(meta, val2str=True, consider_ucn=consider_ucn)
            # inject a few basic properties into the dict
            # analog to what the other modes do in their index
            doc.update({
                k: res[k] for k in ('@id', 'type', 'path', 'parentds')
                if k in res})
            # use search instead of match to not just get hits at the start of the string
            # this will be slower, but avoids having to use actual regex syntax at the user
            # side even for simple queries
            # DOTALL is needed to handle multiline description fields and such, and still
            # be able to match content coming for a later field
            lgr.log(7, "Querying %s among %d items", query, len(doc))
            t0 = time()
            matches = {(q['query'] if isinstance(q, dict) else q, k):
                       q['query'].search(v) if isinstance(q, dict) else q.search(v)
                       for k, v in doc.items()
                       for q in query
                       if not isinstance(q, dict) or q['field'].match(k)}
            dt = time() - t0
            lgr.log(7, "Finished querying in %f sec", dt)
            # retain what actually matched
            matched = {k[1]: match.group() for k, match in matches.items() if match}
            # implement AND behavior across query expressions, but OR behavior
            # across queries matching multiple fields for a single query expression
            # for multiple queries, this makes it consistent with a query that
            # has no field specification
            if matched and len(query) == len(set(k[0] for k in matches if matches[k])):
                hit = dict(
                    res,
                    action='search',
                    query_matched=matched,
                )
                yield hit
                nhits += 1
                if max_nresults and nhits == max_nresults:
                    # report query stats
                    topstr = '{} top {}'.format(
                        max_nresults,
                        single_or_plural('match', 'matches', max_nresults)
                    )
                    lgr.info(
                        "Reached the limit of {}, there could be more which "
                        "were not reported.".format(topstr)
                    )
                    break

    def show_keys(self, mode=None, regexes=None):
        """

        Parameters
        ----------
        mode: {"name", "short", "full"}
        regexes: list of regex
          Which keys to bother working on
        """
        maxl = 100  # approx maximal line length for unique values in mode=short
        # use a dict already, later we need to map to a definition
        # meanwhile map to the values

        keys = self._get_keys(mode)

        for k in sorted(keys):
            if regexes and not self._key_matches(k, regexes):
                continue
            if mode == 'name':
                print(k)
                continue

            # do a bit more
            stat = keys[k]
            all_uvals = uvals = sorted(stat.uvals)

            stat.uvals_str = ensure_unicode(
                "{} unique values: ".format(len(all_uvals))
            )

            if mode == 'short':
                # show only up until we fill maxl
                uvals_str = ''
                uvals = []
                for v in all_uvals:
                    appendix = ('; ' if uvals else '') + v
                    if len(uvals_str) + len(appendix) > maxl - len(stat.uvals_str):
                        break
                    uvals.append(v)
                    uvals_str += appendix
            elif mode == 'full':
                pass
            else:
                raise ValueError(
                    "Unknown value for stats. Know full and short")

            stat.uvals_str += '; '.join(uvals)

            if len(all_uvals) > len(uvals):
                stat.uvals_str += \
                    '; +%s' % single_or_plural("value", "values", len(all_uvals) - len(uvals), True)

            print(
                u'{k}\n in  {stat.ndatasets} datasets\n has {stat.uvals_str}'.format(
                k=k, stat=stat
            ))
        # After #2156 datasets may not necessarily carry all
        # keys in the "unique" summary
        lgr.warning('In this search mode, the reported list of metadata keys may be incomplete')


    def _get_keys(self, mode=None):
        """Return keys and their statistics if mode != 'name'."""
        class key_stat:
            def __init__(self):
                self.ndatasets = 0  # how many datasets have this field
                self.uvals = set()

        from collections import defaultdict
        keys = defaultdict(key_stat)
        for res in query_aggregated_metadata(
                # XXX TODO After #2156 datasets may not necessarily carry all
                # keys in the "unique" summary
                reporton='datasets',
                ds=self.ds,
                aps=[dict(path=self.ds.path, type='dataset')],
                recursive=True):
            meta = res.get('metadata', {})
            # inject a few basic properties into the dict
            # analog to what the other modes do in their index
            meta.update({
                k: res.get(k, None) for k in ('@id', 'type', 'path', 'parentds')
                # parentds is tricky all files will have it, but the dataset
                # queried above might not (single dataset), let's force it in
                if k == 'parentds' or k in res})

            # no stringification of values for speed
            idxd = _meta2autofield_dict(meta, val2str=False)

            for k, kvals in idxd.items():
                # TODO deal with conflicting definitions when available
                keys[k].ndatasets += 1
                if mode == 'name':
                    continue
                keys[k].uvals |= self.get_repr_uvalues(kvals)
        return keys

    def get_repr_uvalues(self, kvals):
        kvals_set = set()
        if not kvals:
            return kvals_set
        kvals_iter = (
            kvals
            if hasattr(kvals, '__iter__') and not isinstance(kvals, (str, bytes))
            else [kvals]
        )
        return set(shortened_repr(x, 50) for x in kvals_iter)

    def get_query(self, query):
        query = ensure_list(query)
        simple_fieldspec = re.compile(r"(?P<field>\S*?):(?P<query>.*)")
        quoted_fieldspec = re.compile(r"'(?P<field>[^']+?)':(?P<query>.*)")
        query_rec_matches = [
            simple_fieldspec.match(q) or
            quoted_fieldspec.match(q) or
            q
            for q in query]
        query_group_dicts_only = [
            q.groupdict() for q in query_rec_matches if hasattr(q, 'groupdict')
        ]
        self._queried_keys = [
            qgd['field']
            for qgd in query_group_dicts_only
            if ('field' in qgd and qgd['field'])
        ]
        if len(query_group_dicts_only) != len(query_rec_matches):
            # we had a query element without field specification add
            # None as an indicator of that
            self._queried_keys.append(None)
        # expand matches, compile expressions
        query = [
            {k: self._compile_query(v) for k, v in q.groupdict().items()}
            if hasattr(q, 'groupdict') else self._compile_query(q)
            for q in query_rec_matches
        ]

        # turn "empty" field specs into simple queries
        # this is used to forcibly disable field-based search
        # e.g. when searching for a value
        query = [q['query']
                 if isinstance(q, dict) and q['field'].pattern == '' else q
                 for q in query]
        return query

    def _xfm_query(self, q):
        # implement potential transformations of regex before they get compiled
        return q

    def _compile_query(self, q):
        """xfm and compile the query, with informative exception if query is incorrect
        """
        q_xfmed = self._xfm_query(q)
        try:
            return re.compile(q_xfmed)
        except re.error as exc:
            omsg = " (original: '%s')" % q if q != q_xfmed else ''
            raise ValueError(
                "regular expression '%s'%s is incorrect: %s"
                % (q_xfmed, omsg, exc)
            )

    def get_nohits_msg(self):
        """Given the query and performed search, provide recommendation

        Quite often a key in the query is mistyped or I simply query for something
        which is not actually known.  It requires --show-keys  run first, doing
        visual search etc to mitigate.  Here we can analyze either all queried
        keys are actually known, and if not known -- what would be the ones available.

        Returns
        -------
        str
          A sentence or a paragraph to be logged/output
        """
        #
        queried_keys = self._queried_keys[:]
        if queried_keys and None in queried_keys:
            queried_keys.pop(queried_keys.index(None))
        if not queried_keys:
            return  # No keys were queried, we are of no use here
        known_keys = self._get_keys(mode='name')
        unknown_keys = sorted(list(set(queried_keys).difference(known_keys)))
        if not unknown_keys:
            return  # again we are of no help here
        msg = super(_EGrepCSSearch, self).get_nohits_msg()
        msg += " Following keys were not found in available metadata: %s. " \
              % ", ".join(unknown_keys)
        suggestions_msg = get_suggestions_msg(unknown_keys, known_keys)
        if suggestions_msg:
            msg += ' ' + suggestions_msg
        return msg


class _EGrepSearch(_EGrepCSSearch):
    _mode_label = 'egrep'
    _default_documenttype = 'datasets'

    def _xfm_query(self, q):
        if q == q.lower():
            # we have no upper case symbol in the query, go case-insensitive
            return '(?i){}'.format(q)
        else:
            return q


@build_doc
class Search(Interface):
    """Search dataset metadata

    DataLad can search metadata extracted from a dataset and/or aggregated into
    a superdataset (see the `aggregate-metadata` command). This makes it
    possible to discover datasets, or individual files in a dataset even when
    they are not available locally.

    Ultimately DataLad metadata are a graph of linked data structures. However,
    this command does not (yet) support queries that can exploit all
    information stored in the metadata. At the moment the following search
    modes are implemented that represent different trade-offs between the
    expressiveness of a query and the computational and storage resources
    required to execute a query.

    - egrep (default)

    - egrepcs [case-sensitive egrep]

    - textblob

    - autofield

    An alternative default mode can be configured by tuning the
    configuration variable 'datalad.search.default-mode'::

      [datalad "search"]
        default-mode = egrepcs

    Each search mode has its own default configuration for what kind of
    documents to query. The respective default can be changed via configuration
    variables::

      [datalad "search"]
        index-<mode_name>-documenttype = (all|datasets|files)


    *Mode: egrep/egrepcs*

    These search modes are largely ignorant of the metadata structure, and
    simply perform matching of a search pattern against a flat
    string-representation of metadata. This is advantageous when the query is
    simple and the metadata structure is irrelevant, or precisely known.
    Moreover, it does not require a search index, hence results can be reported
    without an initial latency for building a search index when the underlying
    metadata has changed (e.g. due to a dataset update). By default, these
    search modes only consider datasets and do not investigate records for
    individual files for speed reasons. Search results are reported in the
    order in which they were discovered.

    Queries can make use of Python regular expression syntax
    (https://docs.python.org/3/library/re.html). In `egrep` mode, matching is
    case-insensitive when the query does not contain upper case characters, but
    is case-sensitive when it does. In `egrepcs` mode, matching is always
    case-sensitive. Expressions will match anywhere in a metadata string, not
    only at the start.

    When multiple queries are given, all queries have to match for a search hit
    (AND behavior).

    It is possible to search individual metadata key/value items by prefixing
    the query with a metadata key name, separated by a colon (':'). The key
    name can also be a regular expression to match multiple keys. A query match
    happens when any value of an item with a matching key name matches the query
    (OR behavior). See examples for more information.

    Examples:

      Query for (what happens to be) an author::

        % datalad search haxby

      Queries are case-INsensitive when the query contains no upper case characters,
      and can be regular expressions. Use `egrepcs` mode when it is desired
      to perform a case-sensitive lowercase match::

        % datalad search --mode egrepcs halchenko.*haxby

      This search mode performs NO analysis of the metadata content.  Therefore
      queries can easily fail to match. For example, the above query implicitly
      assumes that authors are listed in alphabetical order.  If that is the
      case (which may or may not be true), the following query would yield NO
      hits::

        % datalad search Haxby.*Halchenko

      The ``textblob`` search mode represents an alternative that is more
      robust in such cases.

      For more complex queries multiple query expressions can be provided that
      all have to match to be considered a hit (AND behavior). This query
      discovers all files (non-default behavior) that match 'bids.type=T1w'
      AND 'nifti1.qform_code=scanner'::

        % datalad -c datalad.search.index-egrep-documenttype=all search bids.type:T1w nifti1.qform_code:scanner

      Key name selectors can also be expressions, which can be used to select
      multiple keys or construct "fuzzy" queries. In such cases a query matches
      when any item with a matching key matches the query (OR behavior).
      However, multiple queries are always evaluated using an AND conjunction.
      The following query extends the example above to match any files that
      have either 'nifti1.qform_code=scanner' or 'nifti1.sform_code=scanner'::

        % datalad -c datalad.search.index-egrep-documenttype=all search bids.type:T1w nifti1.(q|s)form_code:scanner

    *Mode: textblob*

    This search mode is very similar to the ``egrep`` mode, but with a few key
    differences. A search index is built from the string-representation of
    metadata records. By default, only datasets are included in this index, hence
    the indexing is usually completed within a few seconds, even for hundreds
    of datasets. This mode uses its own query language (not regular expressions)
    that is similar to other search engines. It supports logical conjunctions
    and fuzzy search terms. More information on this is available from the Whoosh
    project (search engine implementation):

      - Description of the Whoosh query language:
        http://whoosh.readthedocs.io/en/latest/querylang.html)

      - Description of a number of query language customizations that are
        enabled in DataLad, such as, fuzzy term matching:
        http://whoosh.readthedocs.io/en/latest/parsing.html#common-customizations

    Importantly, search hits are scored and reported in order of descending
    relevance, hence limiting the number of search results is more meaningful
    than in the 'egrep' mode and can also reduce the query duration.

    Examples:

      Search for (what happens to be) two authors, regardless of the order in
      which those names appear in the metadata::

        % datalad search --mode textblob halchenko haxby

      Fuzzy search when you only have an approximate idea what you are looking
      for or how it is spelled::

        % datalad search --mode textblob haxbi~

      Very fuzzy search, when you are basically only confident about the first
      two characters and how it sounds approximately (or more precisely: allow
      for three edits and require matching of the first two characters)::

        % datalad search --mode textblob haksbi~3/2

      Combine fuzzy search with logical constructs::

        % datalad search --mode textblob 'haxbi~ AND (hanke OR halchenko)'


    *Mode: autofield*

    This mode is similar to the 'textblob' mode, but builds a vastly more
    detailed search index that represents individual metadata variables as
    individual fields. By default, this search index includes records for
    datasets and individual fields, hence it can grow very quickly into
    a huge structure that can easily take an hour or more to build and require
    more than a GB of storage. However, limiting it to documents on datasets
    (see above) retains the enhanced expressiveness of queries while
    dramatically reducing the resource demands.

    Examples:

      List names of search index fields (auto-discovered from the set of
      indexed datasets) which either have a field starting with "age" or
      "gender"::

        % datalad search --mode autofield --show-keys name '\.age' '\.gender'

      Fuzzy search for datasets with an author that is specified in a particular
      metadata field::

        % datalad search --mode autofield bids.author:haxbi~ type:dataset

      Search for individual files that carry a particular description
      prefix in their 'nifti1' metadata::

        % datalad search --mode autofield nifti1.description:FSL* type:file


    *Reporting*

    Search hits are returned as standard DataLad results. On the command line
    the '--output-format' (or '-f') option can be used to tweak results for
    further processing.

    Examples:

      Format search hits as a JSON stream (one hit per line)::

        % datalad -f json search haxby

      Custom formatting: which terms matched the query of particular
      results. Useful for investigating fuzzy search results::

        $ datalad -f '{path}: {query_matched}' search --mode autofield bids.author:haxbi~
    """
    _params_ = dict(
        dataset=Parameter(
            args=("-d", "--dataset"),
            doc="""specify the dataset to perform the query operation on. If
            no dataset is given, an attempt is made to identify the dataset
            based on the current working directory and/or the `path` given""",
            constraints=EnsureDataset() | EnsureNone()),
        query=Parameter(
            args=("query",),
            metavar='QUERY',
            nargs="*",
            doc="""query string, supported syntax and features depends on the
            selected search mode (see documentation)"""),
        force_reindex=Parameter(
            args=("--reindex",),
            dest='force_reindex',
            action='store_true',
            doc="""force rebuilding the search index, even if no change in the
            dataset's state has been detected, for example, when the index
            documenttype configuration has changed."""),
        max_nresults=Parameter(
            args=("--max-nresults",),
            doc="""maxmimum number of search results to report. Setting this
            to 0 will report all search matches. Depending on the mode this
            can search substantially slower. If not specified, a
            mode-specific default setting will be used.""",
            constraints=EnsureInt() | EnsureNone()),
        mode=Parameter(
            args=("--mode",),
            choices=('egrep', 'textblob', 'autofield'),
            doc="""Mode of search index structure and content. See section
            SEARCH MODES for details."""),
        full_record=Parameter(
            args=("--full-record", '-f'),
            action='store_true',
            doc="""If set, return the full metadata record for each search hit.
            Depending on the search mode this might require additional queries.
            By default, only data that is available to the respective search modes
            is returned. This always includes essential information, such as the
            path and the type."""),
        show_keys=Parameter(
            args=('--show-keys',),
            choices=('name', 'short', 'full'),
            default=None,
            doc="""if given, a list of known search keys is shown. If 'name' -
            only the name is printed one per line. If 'short' or 'full',
            statistics (in how many datasets, and how many unique values) are
            printed. 'short' truncates the listing of unique values.
            QUERY, if provided, is regular expressions any of which keys should
            contain.
            No other action is performed (except for reindexing), even if other
            arguments are given. Each key is accompanied by a term definition in
            parenthesis (TODO). In most cases a definition is given in the form
            of a URL. If an ontology definition for a term is known, this URL
            can resolve to a webpage that provides a comprehensive definition
            of the term. However, for speed reasons term resolution is solely done
            on information contained in a local dataset's metadata, and definition
            URLs might be outdated or point to no longer existing resources."""),
        show_query=Parameter(
            args=('--show-query',),
            action='store_true',
            doc="""if given, the formal query that was generated from the given
            query string is shown, but not actually executed. This is mostly useful
            for debugging purposes."""),
    )

    @staticmethod
    @datasetmethod(name='search')
    @eval_results
    def __call__(query=None,
                 dataset=None,
                 force_reindex=False,
                 max_nresults=None,
                 mode=None,
                 full_record=False,
                 show_keys=None,
                 show_query=False):
        try:
            ds = require_dataset(dataset, check_installed=True, purpose='dataset search')
            if ds.id is None:
                raise NoDatasetFound(
                    "This does not seem to be a dataset (no DataLad dataset ID "
                    "found). 'datalad create --force %s' can initialize "
                    "this repository as a DataLad dataset" % ds.path)
        except NoDatasetFound:
            for r in _search_from_virgin_install(dataset, query):
                yield r
            return

        if mode is None:
            # let's get inspired by what the dataset/user think is
            # default
            mode = ds.config.obtain('datalad.search.default-mode')

        if mode == 'egrep':
            searcher = _EGrepSearch
        elif mode == 'egrepcs':
            searcher = _EGrepCSSearch
        elif mode == 'textblob':
            searcher = _BlobSearch
        elif mode == 'autofield':
            searcher = _AutofieldSearch
        else:
            raise ValueError(
                'unknown search mode "{}"'.format(mode))

        searcher = searcher(ds, force_reindex=force_reindex)

        if show_keys:
            searcher.show_keys(show_keys, regexes=query)
            return

        if not query:
            return

        if show_query:
            print(repr(searcher.get_query(query)))
            return

        nhits = 0
        for r in searcher(
                query,
                max_nresults=max_nresults,
                full_record=full_record):
            nhits += 1
            yield r
        if not nhits:
            lgr.info(searcher.get_nohits_msg() or 'no hits')

```
--- END FILE ---

Please first localize the bug based on the issue statement, and then generate *SEARCH/REPLACE* edits to fix the issue.

Every *SEARCH/REPLACE* edit must use this format:
1. The file path
2. The start of search block: <<<<<<< SEARCH
3. A contiguous chunk of lines to search for in the existing source code
4. The dividing line: =======
5. The lines to replace into the source code
6. The end of the replace block: >>>>>>> REPLACE

Here is an example:

```python
### mathweb/flask/app.py
<<<<<<< SEARCH
from flask import Flask
=======
import math
from flask import Flask
>>>>>>> REPLACE
```
</solution>

Please note that the *SEARCH/REPLACE* edit REQUIRES PROPER INDENTATION. If you would like to add the line '        print(x)', you must fully write that out, with all those spaces before the code!
Wrap each *SEARCH/REPLACE* edit in a code block as shown in the example above. If you have multiple *SEARCH/REPLACE* edits, use a separate code block for each one.
[ISSUE]

**Title:** External Metadata Indexers Not Invoked, Causing Incomplete Metadata

**Description:**
When using external metadata indexers, the additional metadata they are supposed to provide is missing. Instead of including the key-value pairs from the external indexer, only the original metadata is present.

**Example:**
```python
metadata = {
    'datalad_unique_content_properties': {
        'extr1': {'prop1': 'v1'}
    },
    'extr1': {'prop1': 'value'}
}

processed_metadata = _meta2autofield_dict(metadata)

print(processed_metadata)
# Output: {'extr1.prop1': 'value'}
```

**Expected Behavior:**
The external indexer should process the `extr1` metadata and add the `jubel` and `trubel` fields, resulting in:
```python
{'extr1.jubel': '1', 'extr1.trubel': '2'}
```

**Actual Behavior:**
Only the original `extr1.prop1` key is present in the processed metadata, and the fields from the external indexer are missing.

[/ISSUE]
{
    "test_listdict2dictlist": "PASSED",
    "test_meta2autofield_dict": "PASSED",
    "test_external_indexer": "PASSED",
    "test_faulty_external_indexer": "PASSED",
    "test_search_outside1_noninteractive_ui": "ERROR",
    "test_search_outside1": "ERROR",
    "test_search_outside1_install_default_ds": "ERROR",
    "test_search_non_dataset": "ERROR",
    "test_within_ds_file_search": "FAILED"
}
diff --git a/datalad/metadata/indexers/__init__.py b/datalad/metadata/indexers/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/datalad/metadata/indexers/base.py b/datalad/metadata/indexers/base.py
index 0c015a3e0..27f3a416c 100644
--- a/datalad/metadata/indexers/base.py
+++ b/datalad/metadata/indexers/base.py
@@ -38,7 +38,7 @@ class MetadataIndexer(object):
           Metadata created by an extractor.
 
         Returns
-        ----------
+        -------
         dict:
            key-value pairs representing the information in metadata.
            values can be literals or lists of literals
diff --git a/datalad/metadata/search.py b/datalad/metadata/search.py
index 7698d2474..4e8977424 100644
--- a/datalad/metadata/search.py
+++ b/datalad/metadata/search.py
@@ -25,8 +25,6 @@ from os.path import normpath
 import sys
 from time import time
 
-from pkg_resources import EntryPoint, iter_entry_points
-
 from datalad import cfg
 from datalad.interface.base import Interface
 from datalad.interface.base import build_doc
@@ -175,7 +173,11 @@ def _meta2autofield_dict(meta, val2str=True, schema=None, consider_ucn=True):
             else:
                 yield key, v
 
-    def get_indexer_for(metadata_format_name: str) -> callable:
+    def get_indexer(metadata_format_name: str) -> callable:
+        from pkg_resources import EntryPoint, iter_entry_points
+
+        # Use the first returned entry point, if any, because there should be
+        # at most one indexer for ``metadata_format_name´´.
         indexer = (tuple(iter_entry_points('datalad.metadata.indexers', metadata_format_name)) or (None,))[0]
         if isinstance(indexer, EntryPoint):
             try:
@@ -185,19 +187,21 @@ def _meta2autofield_dict(meta, val2str=True, schema=None, consider_ucn=True):
                 lgr.warning('Failed to load indexer %s: %s', indexer.name, exc_str(e))
         return lambda metadata: _deep_kv('', metadata)
 
-    def _iterable_to_unicode_string(value, perform_operation):
-        if perform_operation:
+    if val2str:
+        def _val2str_helper(value):
             if isinstance(value, (list, tuple)):
                 return u' '.join(_any2unicode(i) for i in value)
             return _any2unicode(value)
-        return value
+    else:
+        def _val2str_helper(value):
+            return value
 
     meta = meta or {}
     return {
         # Collect all meta-items which have a non-dict value type and where
         # the key not absent in a given schema.
         **{
-            key: _iterable_to_unicode_string(value, val2str)
+            key: _val2str_helper(value)
             for key, value in filter(lambda kv: not isinstance(kv[1], dict), meta.items())
             if schema is None or key in schema
         },
@@ -209,12 +213,12 @@ def _meta2autofield_dict(meta, val2str=True, schema=None, consider_ucn=True):
         # be the name of the extractor, i.e. the metadata_format_name, that created
         # the metadata.
         **{
-            metadata_format_name + '.' + sub_key: _iterable_to_unicode_string(sub_key_value, val2str)
+            metadata_format_name + '.' + sub_key: _val2str_helper(sub_key_value)
             for metadata_format_name, metadata_content in filter(
                 lambda kv: isinstance(kv[1], dict) and kv[0] != 'datalad_unique_content_properties',
                 meta.items()
             )
-            for sub_key, sub_key_value in get_indexer_for(metadata_format_name)(metadata_content)
+            for sub_key, sub_key_value in get_indexer(metadata_format_name)(metadata_content)
             if schema is None or metadata_format_name + '.' + sub_key in schema
         }
     }

{"datalad/metadata/search.py": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 noet:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Interface for managing metadata\n\"\"\"\n\n__docformat__ = 'restructuredtext'\n\nimport collections\nimport logging\nfrom datalad.log import log_progress\nlgr = logging.getLogger('datalad.metadata.search')\n\nimport os\nimport re\nfrom functools import partial\nfrom os.path import join as opj, exists\nfrom os.path import relpath\nfrom os.path import normpath\nimport sys\nfrom time import time\n\nfrom pkg_resources import EntryPoint, iter_entry_points\n\nfrom datalad import cfg\nfrom datalad.interface.base import Interface\nfrom datalad.interface.base import build_doc\nfrom datalad.interface.utils import eval_results\nfrom datalad.distribution.dataset import Dataset\nfrom datalad.distribution.dataset import datasetmethod, EnsureDataset, \\\n    require_dataset\nfrom datalad.support.gitrepo import GitRepo\nfrom datalad.support.param import Parameter\nfrom datalad.support.constraints import EnsureNone\nfrom datalad.support.constraints import EnsureInt\n\nfrom datalad.consts import SEARCH_INDEX_DOTGITDIR\nfrom datalad.utils import (\n    as_unicode,\n    ensure_list,\n    ensure_unicode,\n    get_suggestions_msg,\n    shortened_repr,\n    unicode_srctypes,\n)\nfrom datalad.support.exceptions import NoDatasetFound\nfrom datalad.ui import ui\nfrom datalad.dochelpers import single_or_plural\nfrom datalad.dochelpers import exc_str\nfrom datalad.metadata.metadata import query_aggregated_metadata\n\n# TODO: consider using plain as_unicode, without restricting\n# the types?\n_any2unicode = partial(as_unicode, cast_types=(int, float, tuple, list, dict))\n\n\ndef _listdict2dictlist(lst, strict=True):\n    \"\"\"Helper to deal with DataLad's unique value reports\n\n    Parameters\n    ----------\n    lst : list\n      List of dicts\n    strict : bool\n      In strict mode any dictionary items that doesn't have a simple value\n      (but another container) is discarded. In non-strict mode, arbitrary\n      levels of nesting are preserved, and no unique-ification of values\n      is performed. The latter can be used when it is mostly (or merely)\n      interesting to get a list of metadata keys.\n    \"\"\"\n    # unique values that we got, always a list\n    if all(not isinstance(uval, dict) for uval in lst):\n        # no nested structures, take as is\n        return lst\n\n    type_excluder = (tuple, list)\n    if strict:\n        type_excluder += (dict,)\n    # we need to turn them inside out, instead of a list of\n    # dicts, we want a dict where values are lists, because we\n    # cannot handle hierarchies in a tabular search index\n    # if you came here to find that out, go ahead and use a graph\n    # DB/search\n    udict = {}\n    for uv in lst:\n        if not isinstance(uv, dict):\n            # we cannot mix and match, discard any scalar values\n            # in favor of the structured metadata\n            continue\n        for k, v in uv.items():\n            if isinstance(v, type_excluder):\n                # this is where we draw the line, two levels of\n                # nesting. whoosh can only handle string values\n                # injecting a stringified blob of something doesn't\n                # really enable anything useful -> graph search\n                continue\n            if v == \"\":\n                # no cruft\n                continue\n            if strict:\n                # no duplicate values, only hashable stuff\n                uvals = udict.get(k, set())\n                uvals.add(v)\n            else:\n                # whatever it is, we'll take it\n                uvals = udict.get(k, [])\n                uvals.append(v)\n            udict[k] = uvals\n    return {\n        # set not good for JSON, have plain list\n        k: list(v) if len(v) > 1 else list(v)[0]\n        for k, v in udict.items()\n        # do not accumulate cruft\n        if len(v)}\n\n\ndef _meta2autofield_dict(meta, val2str=True, schema=None, consider_ucn=True):\n    \"\"\"Takes care of dtype conversion into unicode, potential key mappings\n    and concatenation of sequence-type fields into CSV strings\n\n    - if `consider_ucn` (default) it would copy keys from\n      datalad_unique_content_properties into `meta` for that extractor\n    - ... TODO ...\n    \"\"\"\n    if consider_ucn:\n        # loop over all metadata sources and the report of their unique values\n        ucnprops = meta.get(\"datalad_unique_content_properties\", {})\n        for src, umeta in ucnprops.items():\n            srcmeta = meta.get(src, {})\n            for uk in umeta:\n                if uk in srcmeta:\n                    # we have a real entry for this key in the dataset metadata\n                    # ignore any generated unique value list in favor of the\n                    # tailored data\n                    continue\n                srcmeta[uk] = _listdict2dictlist(umeta[uk], strict=False) if umeta[uk] is not None else None\n            if src not in meta and srcmeta:\n                meta[src] = srcmeta  # assign the new one back\n\n    srcmeta = None   # for paranoids to avoid some kind of manipulation of the last\n\n    def _deep_kv(basekey, dct):\n        \"\"\"Return key/value pairs of any depth following a rule for key\n        composition\n\n        dct must be a dict\n        \"\"\"\n        for k, v in dct.items():\n            if (k != '@id' and k.startswith('@')) or k == 'datalad_unique_content_properties':\n                # ignore all JSON-LD specials, but @id\n                continue\n            # TODO `k` might need remapping, if another key was already found\n            # with the same definition\n            key = u'{}{}'.format(\n                basekey,\n                # replace now special chars, and avoid spaces\n                # `os.sep` needs to go, because whoosh uses the field name for\n                # temp files during index staging, and trips over absent \"directories\"\n                # TODO maybe even kill parentheses\n                # TODO actually, it might be better to have an explicit whitelist\n                k.lstrip('@').replace(os.sep, '_').replace(' ', '_').replace('-', '_').replace('.', '_').replace(':', '-')\n            )\n            if isinstance(v, list):\n                v = _listdict2dictlist(v)\n\n            if isinstance(v, dict):\n                # dive\n                for i in _deep_kv('{}.'.format(key), v):\n                    yield i\n            else:\n                yield key, v\n\n    def get_indexer_for(metadata_format_name: str) -> callable:\n        indexer = (tuple(iter_entry_points('datalad.metadata.indexers', metadata_format_name)) or (None,))[0]\n        if isinstance(indexer, EntryPoint):\n            try:\n                indexer_object = indexer.load()(metadata_format_name)\n                return indexer_object.create_index\n            except Exception as e:\n                lgr.warning('Failed to load indexer %s: %s', indexer.name, exc_str(e))\n        return lambda metadata: _deep_kv('', metadata)\n\n    def _iterable_to_unicode_string(value, perform_operation):\n        if perform_operation:\n            if isinstance(value, (list, tuple)):\n                return u' '.join(_any2unicode(i) for i in value)\n            return _any2unicode(value)\n        return value\n\n    meta = meta or {}\n    return {\n        # Collect all meta-items which have a non-dict value type and where\n        # the key not absent in a given schema.\n        **{\n            key: _iterable_to_unicode_string(value, val2str)\n            for key, value in filter(lambda kv: not isinstance(kv[1], dict), meta.items())\n            if schema is None or key in schema\n        },\n\n        # Collect all meta-items which have a dict value type and where\n        # the key is neither 'datalad_unique_content_properties' nor absent\n        # in a given schema.\n        # These values are considered as metadata, the keys are considered to\n        # be the name of the extractor, i.e. the metadata_format_name, that created\n        # the metadata.\n        **{\n            metadata_format_name + '.' + sub_key: _iterable_to_unicode_string(sub_key_value, val2str)\n            for metadata_format_name, metadata_content in filter(\n                lambda kv: isinstance(kv[1], dict) and kv[0] != 'datalad_unique_content_properties',\n                meta.items()\n            )\n            for sub_key, sub_key_value in get_indexer_for(metadata_format_name)(metadata_content)\n            if schema is None or metadata_format_name + '.' + sub_key in schema\n        }\n    }\n\n\ndef _search_from_virgin_install(dataset, query):\n    #\n    # this is to be nice to newbies\n    #\n    exc_info = sys.exc_info()\n    if dataset is None:\n        if not ui.is_interactive:\n            raise NoDatasetFound(\n                \"No DataLad dataset found. Specify a dataset to be \"\n                \"searched, or run interactively to get assistance \"\n                \"installing a queriable superdataset.\"\n            )\n        # none was provided so we could ask user whether he possibly wants\n        # to install our beautiful mega-duper-super-dataset?\n        # TODO: following logic could possibly benefit other actions.\n        DEFAULT_DATASET_PATH = cfg.obtain('datalad.locations.default-dataset')\n        if os.path.exists(DEFAULT_DATASET_PATH):\n            default_ds = Dataset(DEFAULT_DATASET_PATH)\n            if default_ds.is_installed():\n                if ui.yesno(\n                    title=\"No DataLad dataset found at current location\",\n                    text=\"Would you like to search the DataLad \"\n                         \"superdataset at %r?\"\n                          % DEFAULT_DATASET_PATH):\n                    pass\n                else:\n                    raise exc_info[1]\n            else:\n                raise NoDatasetFound(\n                    \"No DataLad dataset found at current location. \"\n                    \"The DataLad superdataset location %r exists, \"\n                    \"but does not contain an dataset.\"\n                    % DEFAULT_DATASET_PATH)\n        elif ui.yesno(\n                title=\"No DataLad dataset found at current location\",\n                text=\"Would you like to install the DataLad \"\n                     \"superdataset at %r?\"\n                     % DEFAULT_DATASET_PATH):\n            from datalad.api import install\n            default_ds = install(DEFAULT_DATASET_PATH, source='///')\n            ui.message(\n                \"From now on you can refer to this dataset using the \"\n                \"label '///'\"\n            )\n        else:\n            raise exc_info[1]\n\n        lgr.info(\n            \"Performing search using DataLad superdataset %r\",\n            default_ds.path\n        )\n        for res in default_ds.search(query):\n            yield res\n        return\n    else:\n        raise  # this function is called within exception handling block\n\n\nclass _Search(object):\n    def __init__(self, ds, **kwargs):\n        self.ds = ds\n        self.documenttype = self.ds.config.obtain(\n            'datalad.search.index-{}-documenttype'.format(self._mode_label),\n            default=self._default_documenttype)\n\n    def __call__(self, query, max_nresults=None):\n        raise NotImplementedError\n\n    @classmethod\n    def _key_matches(cls, k, regexes):\n        \"\"\"Return which regex the key matches\n        \"\"\"\n        for regex in regexes:\n            if re.search(regex, k):\n                return regex\n\n    def show_keys(self, *args, **kwargs):\n        raise NotImplementedError(args)\n\n    def get_query(self, query):\n        \"\"\"Prepare query structure specific for a search backend.\n\n        It can also memorize within instance some parameters of the last query\n        which could be used to assist output formatting/structuring later on\n        \"\"\"\n        raise NotImplementedError\n\n    def get_nohits_msg(self):\n        \"\"\"Given what it knows, provide recommendation in the case of no hits\"\"\"\n        return \"No search hits, wrong query? \" \\\n               \"See 'datalad search --show-keys name' for known keys \" \\\n               \"and 'datalad search --help' on how to prepare your query.\"\n\n\nclass _WhooshSearch(_Search):\n    def __init__(self, ds, force_reindex=False, **kwargs):\n        super(_WhooshSearch, self).__init__(ds, **kwargs)\n\n        self.idx_obj = None\n        # where does the bunny have the eggs?\n\n        self.index_dir = opj(str(self.ds.repo.dot_git), SEARCH_INDEX_DOTGITDIR)\n        self._mk_search_index(force_reindex)\n\n    def show_keys(self, mode, regexes=None):\n        \"\"\"\n\n        Parameters\n        ----------\n        mode: {\"name\"}\n        regexes: list of regex\n          Which keys to bother working on\n        \"\"\"\n        if mode != 'name':\n            raise NotImplementedError(\n                \"ATM %s can show only names, so please use show_keys with 'name'\"\n                % self.__class__.__name__\n            )\n        for k in self.idx_obj.schema.names():\n            if regexes and not self._key_matches(k, regexes):\n                continue\n            print(u'{}'.format(k))\n\n    def get_query(self, query):\n        # parse the query string\n        self._mk_parser()\n        # for convenience we accept any number of args-words from the\n        # shell and put them together to a single string here\n        querystr = ' '.join(ensure_list(query))\n        # this gives a formal whoosh query\n        wquery = self.parser.parse(querystr)\n        return wquery\n\n    def _meta2doc(self, meta, val2str=True, schema=None):\n        raise NotImplementedError\n\n    def _mk_schema(self):\n        raise NotImplementedError\n\n    def _mk_parser(self):\n        raise NotImplementedError\n\n    def _mk_search_index(self, force_reindex):\n        \"\"\"Generic entrypoint to index generation\n\n        The actual work that determines the structure and content of the index\n        is done by functions that are passed in as arguments\n\n        `meta2doc` - must return dict for index document from result input\n        \"\"\"\n        from whoosh import index as widx\n        from .metadata import get_ds_aggregate_db_locations\n        dbloc, db_base_path = get_ds_aggregate_db_locations(self.ds)\n        # what is the lastest state of aggregated metadata\n        metadata_state = self.ds.repo.get_last_commit_hexsha(relpath(dbloc, start=self.ds.path))\n        # use location common to all index types, they would all invalidate\n        # simultaneously\n        stamp_fname = opj(self.index_dir, 'datalad_metadata_state')\n        index_dir = opj(self.index_dir, self._mode_label)\n\n        if (not force_reindex) and \\\n                exists(index_dir) and \\\n                exists(stamp_fname) and \\\n                open(stamp_fname).read() == metadata_state:\n            try:\n                # TODO check that the index schema is the same\n                # as the one we would have used for reindexing\n                # TODO support incremental re-indexing, whoosh can do it\n                idx = widx.open_dir(index_dir)\n                lgr.debug(\n                    'Search index contains %i documents',\n                    idx.doc_count())\n                self.idx_obj = idx\n                return\n            except widx.LockError as e:\n                raise e\n            except widx.IndexError as e:\n                # Generic index error.\n                # we try to regenerate\n                lgr.warning(\n                    \"Cannot open existing index %s (%s), will regenerate\",\n                    index_dir, exc_str(e)\n                )\n            except widx.IndexVersionError as e:  # (msg, version, release=None)\n                # Raised when you try to open an index using a format that the\n                # current version of Whoosh cannot read. That is, when the index\n                # you're trying to open is either not backward or forward\n                # compatible with this version of Whoosh.\n                # we try to regenerate\n                lgr.warning(exc_str(e))\n                pass\n            except widx.OutOfDateError as e:\n                # Raised when you try to commit changes to an index which is not\n                # the latest generation.\n                # this should not happen here, but if it does ... KABOOM\n                raise\n            except widx.EmptyIndexError as e:\n                # Raised when you try to work with an index that has no indexed\n                # terms.\n                # we can just continue with generating an index\n                pass\n            except ValueError as e:\n                if 'unsupported pickle protocol' in str(e):\n                    lgr.warning(\n                        \"Cannot open existing index %s (%s), will regenerate\",\n                        index_dir, exc_str(e)\n                    )\n                else:\n                    raise\n\n        lgr.info('{} search index'.format(\n            'Rebuilding' if exists(index_dir) else 'Building'))\n\n        if not exists(index_dir):\n            os.makedirs(index_dir)\n\n        # this is a pretty cheap call that just pull this info from a file\n        dsinfo = self.ds.metadata(\n            get_aggregates=True,\n            return_type='list',\n            result_renderer='disabled')\n\n        self._mk_schema(dsinfo)\n\n        idx_obj = widx.create_in(index_dir, self.schema)\n        idx = idx_obj.writer(\n            # cache size per process\n            limitmb=cfg.obtain('datalad.search.indexercachesize'),\n            # disable parallel indexing for now till #1927 is resolved\n            ## number of processes for indexing\n            #procs=multiprocessing.cpu_count(),\n            ## write separate index segments in each process for speed\n            ## asks for writer.commit(optimize=True)\n            #multisegment=True,\n        )\n\n        # load metadata of the base dataset and what it knows about all its subdatasets\n        # (recursively)\n        old_idx_size = 0\n        old_ds_rpath = ''\n        idx_size = 0\n        log_progress(\n            lgr.info,\n            'autofieldidxbuild',\n            'Start building search index',\n            total=len(dsinfo),\n            label='Building search index',\n            unit=' Datasets',\n        )\n        for res in query_aggregated_metadata(\n                reporton=self.documenttype,\n                ds=self.ds,\n                aps=[dict(path=self.ds.path, type='dataset')],\n                # MIH: I cannot see a case when we would not want recursion (within\n                # the metadata)\n                recursive=True):\n            # this assumes that files are reported after each dataset report,\n            # and after a subsequent dataset report no files for the previous\n            # dataset will be reported again\n            meta = res.get('metadata', {})\n            doc = self._meta2doc(meta)\n            admin = {\n                'type': res['type'],\n                'path': relpath(res['path'], start=self.ds.path),\n            }\n            if 'parentds' in res:\n                admin['parentds'] = relpath(res['parentds'], start=self.ds.path)\n            if admin['type'] == 'dataset':\n                if old_ds_rpath:\n                    lgr.debug(\n                        'Added %s on dataset %s',\n                        single_or_plural(\n                            'document',\n                            'documents',\n                            idx_size - old_idx_size,\n                            include_count=True),\n                        old_ds_rpath)\n                log_progress(lgr.info, 'autofieldidxbuild',\n                             'Indexed dataset at %s', old_ds_rpath,\n                             update=1, increment=True)\n                old_idx_size = idx_size\n                old_ds_rpath = admin['path']\n                admin['id'] = res.get('dsid', None)\n\n            doc.update({k: ensure_unicode(v) for k, v in admin.items()})\n            lgr.debug(\"Adding document to search index: {}\".format(doc))\n            # inject into index\n            idx.add_document(**doc)\n            idx_size += 1\n\n        if old_ds_rpath:\n            lgr.debug(\n                'Added %s on dataset %s',\n                single_or_plural(\n                    'document',\n                    'documents',\n                    idx_size - old_idx_size,\n                    include_count=True),\n                old_ds_rpath)\n\n        lgr.debug(\"Committing index\")\n        idx.commit(optimize=True)\n        log_progress(\n            lgr.info, 'autofieldidxbuild', 'Done building search index')\n\n        # \"timestamp\" the search index to allow for automatic invalidation\n        with open(stamp_fname, 'w') as f:\n            f.write(metadata_state)\n\n        lgr.info('Search index contains %i documents', idx_size)\n        self.idx_obj = idx_obj\n\n    def __call__(self, query, max_nresults=None, force_reindex=False, full_record=False):\n        if max_nresults is None:\n            # mode default\n            max_nresults = 20\n\n        with self.idx_obj.searcher() as searcher:\n            wquery = self.get_query(query)\n\n            # perform the actual search\n            hits = searcher.search(\n                wquery,\n                terms=True,\n                limit=max_nresults if max_nresults > 0 else None)\n            # report query stats\n            topstr = '{} top {}'.format(\n                max_nresults,\n                single_or_plural('match', 'matches', max_nresults)\n            )\n            lgr.info('Query completed in {} sec.{}'.format(\n                hits.runtime,\n                ' Reporting {}.'.format(\n                    ('up to ' + topstr)\n                    if max_nresults > 0\n                    else 'all matches'\n                )\n                if not hits.is_empty()\n                else ' No matches.'\n            ))\n\n            if not hits:\n                return\n\n            nhits = 0\n            annotated_hits = []\n            # annotate hits for full metadata report\n            for i, hit in enumerate(hits):\n                annotated_hit = dict(\n                    path=normpath(opj(self.ds.path, hit['path'])),\n                    query_matched={ensure_unicode(k): ensure_unicode(v)\n                                   if isinstance(v, unicode_srctypes) else v\n                                   for k, v in hit.matched_terms()},\n                    parentds=normpath(\n                        opj(self.ds.path, hit['parentds'])) if 'parentds' in hit else None,\n                    type=hit.get('type', None))\n                if not full_record:\n                    annotated_hit.update(\n                        refds=self.ds.path,\n                        action='search',\n                        status='ok',\n                        logger=lgr,\n                    )\n                    yield annotated_hit\n                else:\n                    annotated_hits.append(annotated_hit)\n                nhits += 1\n            if full_record:\n                for res in query_aggregated_metadata(\n                        # type is taken from hit record\n                        reporton=None,\n                        ds=self.ds,\n                        aps=annotated_hits,\n                        # never recursive, we have direct hits already\n                        recursive=False):\n                    res.update(\n                        refds=self.ds.path,\n                        action='search',\n                        status='ok',\n                        logger=lgr,\n                    )\n                    yield res\n\n            if max_nresults and nhits == max_nresults:\n                lgr.info(\n                    \"Reached the limit of {}, there could be more which \"\n                    \"were not reported.\".format(topstr)\n                )\n\n\nclass _BlobSearch(_WhooshSearch):\n    _mode_label = 'textblob'\n    _default_documenttype = 'datasets'\n\n    def _meta2doc(self, meta):\n        # coerce the entire flattened metadata dict into a comma-separated string\n        # that also includes the keys\n        return dict(meta=u', '.join(\n            u'{}: {}'.format(k, v)\n            for k, v in _meta2autofield_dict(\n                meta,\n                val2str=True,\n                schema=None).items()))\n\n    def _mk_schema(self, dsinfo):\n        from whoosh import fields as wf\n        from whoosh.analysis import StandardAnalyzer\n\n        # TODO support some customizable mapping to homogenize some metadata fields\n        # onto a given set of index keys\n        self.schema = wf.Schema(\n            id=wf.ID,\n            path=wf.ID(stored=True),\n            type=wf.ID(stored=True),\n            parentds=wf.ID(stored=True),\n            meta=wf.TEXT(\n                stored=False,\n                analyzer=StandardAnalyzer(minsize=2))\n        )\n\n    def _mk_parser(self):\n        from whoosh import qparser as qparse\n\n        # use whoosh default query parser for now\n        parser = qparse.QueryParser(\n            \"meta\",\n            schema=self.idx_obj.schema\n        )\n        parser.add_plugin(qparse.FuzzyTermPlugin())\n        parser.remove_plugin_class(qparse.PhrasePlugin)\n        parser.add_plugin(qparse.SequencePlugin())\n        self.parser = parser\n\n\nclass _AutofieldSearch(_WhooshSearch):\n    _mode_label = 'autofield'\n    _default_documenttype = 'all'\n\n    def _meta2doc(self, meta):\n        return _meta2autofield_dict(meta, val2str=True, schema=self.schema)\n\n    def _mk_schema(self, dsinfo):\n        from whoosh import fields as wf\n        from whoosh.analysis import SimpleAnalyzer\n\n        # haven for terms that have been found to be undefined\n        # (for faster decision-making upon next encounter)\n        # this will harvest all discovered term definitions\n        definitions = {\n            '@id': 'unique identifier of an entity',\n            # TODO make proper JSON-LD definition\n            'path': 'path name of an entity relative to the searched base dataset',\n            # TODO make proper JSON-LD definition\n            'parentds': 'path of the datasets that contains an entity',\n            # 'type' will not come from a metadata field, hence will not be detected\n            'type': 'type of a record',\n        }\n\n        schema_fields = {\n            n.lstrip('@'): wf.ID(stored=True, unique=n == '@id')\n            for n in definitions}\n\n        lgr.debug('Scanning for metadata keys')\n        # quick 1st pass over all dataset to gather the needed schema fields\n        log_progress(\n            lgr.info,\n            'idxschemabuild',\n            'Start building search schema',\n            total=len(dsinfo),\n            label='Building search schema',\n            unit=' Datasets',\n        )\n        for res in query_aggregated_metadata(\n                # XXX TODO After #2156 datasets may not necessarily carry all\n                # keys in the \"unique\" summary\n                reporton='datasets',\n                ds=self.ds,\n                aps=[dict(path=self.ds.path, type='dataset')],\n                recursive=True):\n            meta = res.get('metadata', {})\n            # no stringification of values for speed, we do not need/use the\n            # actual values at this point, only the keys\n            idxd = _meta2autofield_dict(meta, val2str=False)\n\n            for k in idxd:\n                schema_fields[k] = wf.TEXT(stored=False,\n                                           analyzer=SimpleAnalyzer())\n            log_progress(lgr.info, 'idxschemabuild',\n                         'Scanned dataset at %s', res['path'],\n                         update=1, increment=True)\n        log_progress(\n            lgr.info, 'idxschemabuild', 'Done building search schema')\n\n        self.schema = wf.Schema(**schema_fields)\n\n    def _mk_parser(self):\n        from whoosh import qparser as qparse\n\n        parser = qparse.MultifieldParser(\n            self.idx_obj.schema.names(),\n            self.idx_obj.schema)\n        # XXX: plugin is broken in Debian's whoosh 2.7.0-2, but already fixed\n        # upstream\n        parser.add_plugin(qparse.FuzzyTermPlugin())\n        parser.add_plugin(qparse.GtLtPlugin())\n        parser.add_plugin(qparse.SingleQuotePlugin())\n        # replace field defintion to allow for colons to be part of a field's name:\n        parser.replace_plugin(qparse.FieldsPlugin(expr=r\"(?P<text>[()<>.\\w]+|[*]):\"))\n        self.parser = parser\n\n\nclass _EGrepCSSearch(_Search):\n    _mode_label = 'egrepcs'\n    _default_documenttype = 'datasets'\n\n    def __init__(self, ds, **kwargs):\n        super(_EGrepCSSearch, self).__init__(ds, **kwargs)\n        self._queried_keys = None  # to be memoized by get_query\n\n    # If there were custom \"per-search engine\" options, we could expose\n    # --consider_ucn - search through unique content properties of the dataset\n    #    which might be more computationally demanding\n    def __call__(self, query, max_nresults=None, consider_ucn=False, full_record=True):\n        if max_nresults is None:\n            # no limit by default\n            max_nresults = 0\n        query = self.get_query(query)\n\n        nhits = 0\n        for res in query_aggregated_metadata(\n                reporton=self.documenttype,\n                ds=self.ds,\n                aps=[dict(path=self.ds.path, type='dataset')],\n                # MIH: I cannot see a case when we would not want recursion (within\n                # the metadata)\n                recursive=True):\n            # this assumes that files are reported after each dataset report,\n            # and after a subsequent dataset report no files for the previous\n            # dataset will be reported again\n            meta = res.get('metadata', {})\n            # produce a flattened metadata dict to search through\n            doc = _meta2autofield_dict(meta, val2str=True, consider_ucn=consider_ucn)\n            # inject a few basic properties into the dict\n            # analog to what the other modes do in their index\n            doc.update({\n                k: res[k] for k in ('@id', 'type', 'path', 'parentds')\n                if k in res})\n            # use search instead of match to not just get hits at the start of the string\n            # this will be slower, but avoids having to use actual regex syntax at the user\n            # side even for simple queries\n            # DOTALL is needed to handle multiline description fields and such, and still\n            # be able to match content coming for a later field\n            lgr.log(7, \"Querying %s among %d items\", query, len(doc))\n            t0 = time()\n            matches = {(q['query'] if isinstance(q, dict) else q, k):\n                       q['query'].search(v) if isinstance(q, dict) else q.search(v)\n                       for k, v in doc.items()\n                       for q in query\n                       if not isinstance(q, dict) or q['field'].match(k)}\n            dt = time() - t0\n            lgr.log(7, \"Finished querying in %f sec\", dt)\n            # retain what actually matched\n            matched = {k[1]: match.group() for k, match in matches.items() if match}\n            # implement AND behavior across query expressions, but OR behavior\n            # across queries matching multiple fields for a single query expression\n            # for multiple queries, this makes it consistent with a query that\n            # has no field specification\n            if matched and len(query) == len(set(k[0] for k in matches if matches[k])):\n                hit = dict(\n                    res,\n                    action='search',\n                    query_matched=matched,\n                )\n                yield hit\n                nhits += 1\n                if max_nresults and nhits == max_nresults:\n                    # report query stats\n                    topstr = '{} top {}'.format(\n                        max_nresults,\n                        single_or_plural('match', 'matches', max_nresults)\n                    )\n                    lgr.info(\n                        \"Reached the limit of {}, there could be more which \"\n                        \"were not reported.\".format(topstr)\n                    )\n                    break\n\n    def show_keys(self, mode=None, regexes=None):\n        \"\"\"\n\n        Parameters\n        ----------\n        mode: {\"name\", \"short\", \"full\"}\n        regexes: list of regex\n          Which keys to bother working on\n        \"\"\"\n        maxl = 100  # approx maximal line length for unique values in mode=short\n        # use a dict already, later we need to map to a definition\n        # meanwhile map to the values\n\n        keys = self._get_keys(mode)\n\n        for k in sorted(keys):\n            if regexes and not self._key_matches(k, regexes):\n                continue\n            if mode == 'name':\n                print(k)\n                continue\n\n            # do a bit more\n            stat = keys[k]\n            all_uvals = uvals = sorted(stat.uvals)\n\n            stat.uvals_str = ensure_unicode(\n                \"{} unique values: \".format(len(all_uvals))\n            )\n\n            if mode == 'short':\n                # show only up until we fill maxl\n                uvals_str = ''\n                uvals = []\n                for v in all_uvals:\n                    appendix = ('; ' if uvals else '') + v\n                    if len(uvals_str) + len(appendix) > maxl - len(stat.uvals_str):\n                        break\n                    uvals.append(v)\n                    uvals_str += appendix\n            elif mode == 'full':\n                pass\n            else:\n                raise ValueError(\n                    \"Unknown value for stats. Know full and short\")\n\n            stat.uvals_str += '; '.join(uvals)\n\n            if len(all_uvals) > len(uvals):\n                stat.uvals_str += \\\n                    '; +%s' % single_or_plural(\"value\", \"values\", len(all_uvals) - len(uvals), True)\n\n            print(\n                u'{k}\\n in  {stat.ndatasets} datasets\\n has {stat.uvals_str}'.format(\n                k=k, stat=stat\n            ))\n        # After #2156 datasets may not necessarily carry all\n        # keys in the \"unique\" summary\n        lgr.warning('In this search mode, the reported list of metadata keys may be incomplete')\n\n\n    def _get_keys(self, mode=None):\n        \"\"\"Return keys and their statistics if mode != 'name'.\"\"\"\n        class key_stat:\n            def __init__(self):\n                self.ndatasets = 0  # how many datasets have this field\n                self.uvals = set()\n\n        from collections import defaultdict\n        keys = defaultdict(key_stat)\n        for res in query_aggregated_metadata(\n                # XXX TODO After #2156 datasets may not necessarily carry all\n                # keys in the \"unique\" summary\n                reporton='datasets',\n                ds=self.ds,\n                aps=[dict(path=self.ds.path, type='dataset')],\n                recursive=True):\n            meta = res.get('metadata', {})\n            # inject a few basic properties into the dict\n            # analog to what the other modes do in their index\n            meta.update({\n                k: res.get(k, None) for k in ('@id', 'type', 'path', 'parentds')\n                # parentds is tricky all files will have it, but the dataset\n                # queried above might not (single dataset), let's force it in\n                if k == 'parentds' or k in res})\n\n            # no stringification of values for speed\n            idxd = _meta2autofield_dict(meta, val2str=False)\n\n            for k, kvals in idxd.items():\n                # TODO deal with conflicting definitions when available\n                keys[k].ndatasets += 1\n                if mode == 'name':\n                    continue\n                keys[k].uvals |= self.get_repr_uvalues(kvals)\n        return keys\n\n    def get_repr_uvalues(self, kvals):\n        kvals_set = set()\n        if not kvals:\n            return kvals_set\n        kvals_iter = (\n            kvals\n            if hasattr(kvals, '__iter__') and not isinstance(kvals, (str, bytes))\n            else [kvals]\n        )\n        return set(shortened_repr(x, 50) for x in kvals_iter)\n\n    def get_query(self, query):\n        query = ensure_list(query)\n        simple_fieldspec = re.compile(r\"(?P<field>\\S*?):(?P<query>.*)\")\n        quoted_fieldspec = re.compile(r\"'(?P<field>[^']+?)':(?P<query>.*)\")\n        query_rec_matches = [\n            simple_fieldspec.match(q) or\n            quoted_fieldspec.match(q) or\n            q\n            for q in query]\n        query_group_dicts_only = [\n            q.groupdict() for q in query_rec_matches if hasattr(q, 'groupdict')\n        ]\n        self._queried_keys = [\n            qgd['field']\n            for qgd in query_group_dicts_only\n            if ('field' in qgd and qgd['field'])\n        ]\n        if len(query_group_dicts_only) != len(query_rec_matches):\n            # we had a query element without field specification add\n            # None as an indicator of that\n            self._queried_keys.append(None)\n        # expand matches, compile expressions\n        query = [\n            {k: self._compile_query(v) for k, v in q.groupdict().items()}\n            if hasattr(q, 'groupdict') else self._compile_query(q)\n            for q in query_rec_matches\n        ]\n\n        # turn \"empty\" field specs into simple queries\n        # this is used to forcibly disable field-based search\n        # e.g. when searching for a value\n        query = [q['query']\n                 if isinstance(q, dict) and q['field'].pattern == '' else q\n                 for q in query]\n        return query\n\n    def _xfm_query(self, q):\n        # implement potential transformations of regex before they get compiled\n        return q\n\n    def _compile_query(self, q):\n        \"\"\"xfm and compile the query, with informative exception if query is incorrect\n        \"\"\"\n        q_xfmed = self._xfm_query(q)\n        try:\n            return re.compile(q_xfmed)\n        except re.error as exc:\n            omsg = \" (original: '%s')\" % q if q != q_xfmed else ''\n            raise ValueError(\n                \"regular expression '%s'%s is incorrect: %s\"\n                % (q_xfmed, omsg, exc)\n            )\n\n    def get_nohits_msg(self):\n        \"\"\"Given the query and performed search, provide recommendation\n\n        Quite often a key in the query is mistyped or I simply query for something\n        which is not actually known.  It requires --show-keys  run first, doing\n        visual search etc to mitigate.  Here we can analyze either all queried\n        keys are actually known, and if not known -- what would be the ones available.\n\n        Returns\n        -------\n        str\n          A sentence or a paragraph to be logged/output\n        \"\"\"\n        #\n        queried_keys = self._queried_keys[:]\n        if queried_keys and None in queried_keys:\n            queried_keys.pop(queried_keys.index(None))\n        if not queried_keys:\n            return  # No keys were queried, we are of no use here\n        known_keys = self._get_keys(mode='name')\n        unknown_keys = sorted(list(set(queried_keys).difference(known_keys)))\n        if not unknown_keys:\n            return  # again we are of no help here\n        msg = super(_EGrepCSSearch, self).get_nohits_msg()\n        msg += \" Following keys were not found in available metadata: %s. \" \\\n              % \", \".join(unknown_keys)\n        suggestions_msg = get_suggestions_msg(unknown_keys, known_keys)\n        if suggestions_msg:\n            msg += ' ' + suggestions_msg\n        return msg\n\n\nclass _EGrepSearch(_EGrepCSSearch):\n    _mode_label = 'egrep'\n    _default_documenttype = 'datasets'\n\n    def _xfm_query(self, q):\n        if q == q.lower():\n            # we have no upper case symbol in the query, go case-insensitive\n            return '(?i){}'.format(q)\n        else:\n            return q\n\n\n@build_doc\nclass Search(Interface):\n    \"\"\"Search dataset metadata\n\n    DataLad can search metadata extracted from a dataset and/or aggregated into\n    a superdataset (see the `aggregate-metadata` command). This makes it\n    possible to discover datasets, or individual files in a dataset even when\n    they are not available locally.\n\n    Ultimately DataLad metadata are a graph of linked data structures. However,\n    this command does not (yet) support queries that can exploit all\n    information stored in the metadata. At the moment the following search\n    modes are implemented that represent different trade-offs between the\n    expressiveness of a query and the computational and storage resources\n    required to execute a query.\n\n    - egrep (default)\n\n    - egrepcs [case-sensitive egrep]\n\n    - textblob\n\n    - autofield\n\n    An alternative default mode can be configured by tuning the\n    configuration variable 'datalad.search.default-mode'::\n\n      [datalad \"search\"]\n        default-mode = egrepcs\n\n    Each search mode has its own default configuration for what kind of\n    documents to query. The respective default can be changed via configuration\n    variables::\n\n      [datalad \"search\"]\n        index-<mode_name>-documenttype = (all|datasets|files)\n\n\n    *Mode: egrep/egrepcs*\n\n    These search modes are largely ignorant of the metadata structure, and\n    simply perform matching of a search pattern against a flat\n    string-representation of metadata. This is advantageous when the query is\n    simple and the metadata structure is irrelevant, or precisely known.\n    Moreover, it does not require a search index, hence results can be reported\n    without an initial latency for building a search index when the underlying\n    metadata has changed (e.g. due to a dataset update). By default, these\n    search modes only consider datasets and do not investigate records for\n    individual files for speed reasons. Search results are reported in the\n    order in which they were discovered.\n\n    Queries can make use of Python regular expression syntax\n    (https://docs.python.org/3/library/re.html). In `egrep` mode, matching is\n    case-insensitive when the query does not contain upper case characters, but\n    is case-sensitive when it does. In `egrepcs` mode, matching is always\n    case-sensitive. Expressions will match anywhere in a metadata string, not\n    only at the start.\n\n    When multiple queries are given, all queries have to match for a search hit\n    (AND behavior).\n\n    It is possible to search individual metadata key/value items by prefixing\n    the query with a metadata key name, separated by a colon (':'). The key\n    name can also be a regular expression to match multiple keys. A query match\n    happens when any value of an item with a matching key name matches the query\n    (OR behavior). See examples for more information.\n\n    Examples:\n\n      Query for (what happens to be) an author::\n\n        % datalad search haxby\n\n      Queries are case-INsensitive when the query contains no upper case characters,\n      and can be regular expressions. Use `egrepcs` mode when it is desired\n      to perform a case-sensitive lowercase match::\n\n        % datalad search --mode egrepcs halchenko.*haxby\n\n      This search mode performs NO analysis of the metadata content.  Therefore\n      queries can easily fail to match. For example, the above query implicitly\n      assumes that authors are listed in alphabetical order.  If that is the\n      case (which may or may not be true), the following query would yield NO\n      hits::\n\n        % datalad search Haxby.*Halchenko\n\n      The ``textblob`` search mode represents an alternative that is more\n      robust in such cases.\n\n      For more complex queries multiple query expressions can be provided that\n      all have to match to be considered a hit (AND behavior). This query\n      discovers all files (non-default behavior) that match 'bids.type=T1w'\n      AND 'nifti1.qform_code=scanner'::\n\n        % datalad -c datalad.search.index-egrep-documenttype=all search bids.type:T1w nifti1.qform_code:scanner\n\n      Key name selectors can also be expressions, which can be used to select\n      multiple keys or construct \"fuzzy\" queries. In such cases a query matches\n      when any item with a matching key matches the query (OR behavior).\n      However, multiple queries are always evaluated using an AND conjunction.\n      The following query extends the example above to match any files that\n      have either 'nifti1.qform_code=scanner' or 'nifti1.sform_code=scanner'::\n\n        % datalad -c datalad.search.index-egrep-documenttype=all search bids.type:T1w nifti1.(q|s)form_code:scanner\n\n    *Mode: textblob*\n\n    This search mode is very similar to the ``egrep`` mode, but with a few key\n    differences. A search index is built from the string-representation of\n    metadata records. By default, only datasets are included in this index, hence\n    the indexing is usually completed within a few seconds, even for hundreds\n    of datasets. This mode uses its own query language (not regular expressions)\n    that is similar to other search engines. It supports logical conjunctions\n    and fuzzy search terms. More information on this is available from the Whoosh\n    project (search engine implementation):\n\n      - Description of the Whoosh query language:\n        http://whoosh.readthedocs.io/en/latest/querylang.html)\n\n      - Description of a number of query language customizations that are\n        enabled in DataLad, such as, fuzzy term matching:\n        http://whoosh.readthedocs.io/en/latest/parsing.html#common-customizations\n\n    Importantly, search hits are scored and reported in order of descending\n    relevance, hence limiting the number of search results is more meaningful\n    than in the 'egrep' mode and can also reduce the query duration.\n\n    Examples:\n\n      Search for (what happens to be) two authors, regardless of the order in\n      which those names appear in the metadata::\n\n        % datalad search --mode textblob halchenko haxby\n\n      Fuzzy search when you only have an approximate idea what you are looking\n      for or how it is spelled::\n\n        % datalad search --mode textblob haxbi~\n\n      Very fuzzy search, when you are basically only confident about the first\n      two characters and how it sounds approximately (or more precisely: allow\n      for three edits and require matching of the first two characters)::\n\n        % datalad search --mode textblob haksbi~3/2\n\n      Combine fuzzy search with logical constructs::\n\n        % datalad search --mode textblob 'haxbi~ AND (hanke OR halchenko)'\n\n\n    *Mode: autofield*\n\n    This mode is similar to the 'textblob' mode, but builds a vastly more\n    detailed search index that represents individual metadata variables as\n    individual fields. By default, this search index includes records for\n    datasets and individual fields, hence it can grow very quickly into\n    a huge structure that can easily take an hour or more to build and require\n    more than a GB of storage. However, limiting it to documents on datasets\n    (see above) retains the enhanced expressiveness of queries while\n    dramatically reducing the resource demands.\n\n    Examples:\n\n      List names of search index fields (auto-discovered from the set of\n      indexed datasets) which either have a field starting with \"age\" or\n      \"gender\"::\n\n        % datalad search --mode autofield --show-keys name '\\.age' '\\.gender'\n\n      Fuzzy search for datasets with an author that is specified in a particular\n      metadata field::\n\n        % datalad search --mode autofield bids.author:haxbi~ type:dataset\n\n      Search for individual files that carry a particular description\n      prefix in their 'nifti1' metadata::\n\n        % datalad search --mode autofield nifti1.description:FSL* type:file\n\n\n    *Reporting*\n\n    Search hits are returned as standard DataLad results. On the command line\n    the '--output-format' (or '-f') option can be used to tweak results for\n    further processing.\n\n    Examples:\n\n      Format search hits as a JSON stream (one hit per line)::\n\n        % datalad -f json search haxby\n\n      Custom formatting: which terms matched the query of particular\n      results. Useful for investigating fuzzy search results::\n\n        $ datalad -f '{path}: {query_matched}' search --mode autofield bids.author:haxbi~\n    \"\"\"\n    _params_ = dict(\n        dataset=Parameter(\n            args=(\"-d\", \"--dataset\"),\n            doc=\"\"\"specify the dataset to perform the query operation on. If\n            no dataset is given, an attempt is made to identify the dataset\n            based on the current working directory and/or the `path` given\"\"\",\n            constraints=EnsureDataset() | EnsureNone()),\n        query=Parameter(\n            args=(\"query\",),\n            metavar='QUERY',\n            nargs=\"*\",\n            doc=\"\"\"query string, supported syntax and features depends on the\n            selected search mode (see documentation)\"\"\"),\n        force_reindex=Parameter(\n            args=(\"--reindex\",),\n            dest='force_reindex',\n            action='store_true',\n            doc=\"\"\"force rebuilding the search index, even if no change in the\n            dataset's state has been detected, for example, when the index\n            documenttype configuration has changed.\"\"\"),\n        max_nresults=Parameter(\n            args=(\"--max-nresults\",),\n            doc=\"\"\"maxmimum number of search results to report. Setting this\n            to 0 will report all search matches. Depending on the mode this\n            can search substantially slower. If not specified, a\n            mode-specific default setting will be used.\"\"\",\n            constraints=EnsureInt() | EnsureNone()),\n        mode=Parameter(\n            args=(\"--mode\",),\n            choices=('egrep', 'textblob', 'autofield'),\n            doc=\"\"\"Mode of search index structure and content. See section\n            SEARCH MODES for details.\"\"\"),\n        full_record=Parameter(\n            args=(\"--full-record\", '-f'),\n            action='store_true',\n            doc=\"\"\"If set, return the full metadata record for each search hit.\n            Depending on the search mode this might require additional queries.\n            By default, only data that is available to the respective search modes\n            is returned. This always includes essential information, such as the\n            path and the type.\"\"\"),\n        show_keys=Parameter(\n            args=('--show-keys',),\n            choices=('name', 'short', 'full'),\n            default=None,\n            doc=\"\"\"if given, a list of known search keys is shown. If 'name' -\n            only the name is printed one per line. If 'short' or 'full',\n            statistics (in how many datasets, and how many unique values) are\n            printed. 'short' truncates the listing of unique values.\n            QUERY, if provided, is regular expressions any of which keys should\n            contain.\n            No other action is performed (except for reindexing), even if other\n            arguments are given. Each key is accompanied by a term definition in\n            parenthesis (TODO). In most cases a definition is given in the form\n            of a URL. If an ontology definition for a term is known, this URL\n            can resolve to a webpage that provides a comprehensive definition\n            of the term. However, for speed reasons term resolution is solely done\n            on information contained in a local dataset's metadata, and definition\n            URLs might be outdated or point to no longer existing resources.\"\"\"),\n        show_query=Parameter(\n            args=('--show-query',),\n            action='store_true',\n            doc=\"\"\"if given, the formal query that was generated from the given\n            query string is shown, but not actually executed. This is mostly useful\n            for debugging purposes.\"\"\"),\n    )\n\n    @staticmethod\n    @datasetmethod(name='search')\n    @eval_results\n    def __call__(query=None,\n                 dataset=None,\n                 force_reindex=False,\n                 max_nresults=None,\n                 mode=None,\n                 full_record=False,\n                 show_keys=None,\n                 show_query=False):\n        try:\n            ds = require_dataset(dataset, check_installed=True, purpose='dataset search')\n            if ds.id is None:\n                raise NoDatasetFound(\n                    \"This does not seem to be a dataset (no DataLad dataset ID \"\n                    \"found). 'datalad create --force %s' can initialize \"\n                    \"this repository as a DataLad dataset\" % ds.path)\n        except NoDatasetFound:\n            for r in _search_from_virgin_install(dataset, query):\n                yield r\n            return\n\n        if mode is None:\n            # let's get inspired by what the dataset/user think is\n            # default\n            mode = ds.config.obtain('datalad.search.default-mode')\n\n        if mode == 'egrep':\n            searcher = _EGrepSearch\n        elif mode == 'egrepcs':\n            searcher = _EGrepCSSearch\n        elif mode == 'textblob':\n            searcher = _BlobSearch\n        elif mode == 'autofield':\n            searcher = _AutofieldSearch\n        else:\n            raise ValueError(\n                'unknown search mode \"{}\"'.format(mode))\n\n        searcher = searcher(ds, force_reindex=force_reindex)\n\n        if show_keys:\n            searcher.show_keys(show_keys, regexes=query)\n            return\n\n        if not query:\n            return\n\n        if show_query:\n            print(repr(searcher.get_query(query)))\n            return\n\n        nhits = 0\n        for r in searcher(\n                query,\n                max_nresults=max_nresults,\n                full_record=full_record):\n            nhits += 1\n            yield r\n        if not nhits:\n            lgr.info(searcher.get_nohits_msg() or 'no hits')\n"}
A user will ask you to solve a task. You should first draft your thinking process (inner monologue). Then, generate the solution.

Your response format must follow the template below:
<think>
Your thoughts or/and draft, like working through an exercise on scratch paper. Be as casual and as long as you want until you are confident to generate a correct solution.
</think>
<solution>
Final solution presented to the user.
</solution>
system
We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
[ISSUE]

**Title:** External Metadata Indexers Not Invoked, Causing Incomplete Metadata

**Description:**
When using external metadata indexers, the additional metadata they are supposed to provide is missing. Instead of including the key-value pairs from the external indexer, only the original metadata is present.

**Example:**
```python
metadata = {
    'datalad_unique_content_properties': {
        'extr1': {'prop1': 'v1'}
    },
    'extr1': {'prop1': 'value'}
}

processed_metadata = _meta2autofield_dict(metadata)

print(processed_metadata)
# Output: {'extr1.prop1': 'value'}
```

**Expected Behavior:**
The external indexer should process the `extr1` metadata and add the `jubel` and `trubel` fields, resulting in:
```python
{'extr1.jubel': '1', 'extr1.trubel': '2'}
```

**Actual Behavior:**
Only the original `extr1.prop1` key is present in the processed metadata, and the fields from the external indexer are missing.

[/ISSUE]
--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.

--- BEGIN FILE ---
```
### datalad/metadata/search.py
# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-
# ex: set sts=4 ts=4 sw=4 noet:
# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
#
#   See COPYING file distributed along with the datalad package for the
#   copyright and license terms.
#
# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
"""Interface for managing metadata
"""

__docformat__ = 'restructuredtext'

import collections
import logging
from datalad.log import log_progress
lgr = logging.getLogger('datalad.metadata.search')

import os
import re
from functools import partial
from os.path import join as opj, exists
from os.path import relpath
from os.path import normpath
import sys
from time import time

from pkg_resources import EntryPoint, iter_entry_points

from datalad import cfg
from datalad.interface.base import Interface
from datalad.interface.base import build_doc
from datalad.interface.utils import eval_results
from datalad.distribution.dataset import Dataset
from datalad.distribution.dataset import datasetmethod, EnsureDataset, \
    require_dataset
from datalad.support.gitrepo import GitRepo
from datalad.support.param import Parameter
from datalad.support.constraints import EnsureNone
from datalad.support.constraints import EnsureInt

from datalad.consts import SEARCH_INDEX_DOTGITDIR
from datalad.utils import (
    as_unicode,
    ensure_list,
    ensure_unicode,
    get_suggestions_msg,
    shortened_repr,
    unicode_srctypes,
)
from datalad.support.exceptions import NoDatasetFound
from datalad.ui import ui
from datalad.dochelpers import single_or_plural
from datalad.dochelpers import exc_str
from datalad.metadata.metadata import query_aggregated_metadata

# TODO: consider using plain as_unicode, without restricting
# the types?
_any2unicode = partial(as_unicode, cast_types=(int, float, tuple, list, dict))


def _listdict2dictlist(lst, strict=True):
    """Helper to deal with DataLad's unique value reports

    Parameters
    ----------
    lst : list
      List of dicts
    strict : bool
      In strict mode any dictionary items that doesn't have a simple value
      (but another container) is discarded. In non-strict mode, arbitrary
      levels of nesting are preserved, and no unique-ification of values
      is performed. The latter can be used when it is mostly (or merely)
      interesting to get a list of metadata keys.
    """
    # unique values that we got, always a list
    if all(not isinstance(uval, dict) for uval in lst):
        # no nested structures, take as is
        return lst

    type_excluder = (tuple, list)
    if strict:
        type_excluder += (dict,)
    # we need to turn them inside out, instead of a list of
    # dicts, we want a dict where values are lists, because we
    # cannot handle hierarchies in a tabular search index
    # if you came here to find that out, go ahead and use a graph
    # DB/search
    udict = {}
    for uv in lst:
        if not isinstance(uv, dict):
            # we cannot mix and match, discard any scalar values
            # in favor of the structured metadata
            continue
        for k, v in uv.items():
            if isinstance(v, type_excluder):
                # this is where we draw the line, two levels of
                # nesting. whoosh can only handle string values
                # injecting a stringified blob of something doesn't
                # really enable anything useful -> graph search
                continue
            if v == "":
                # no cruft
                continue
            if strict:
                # no duplicate values, only hashable stuff
                uvals = udict.get(k, set())
                uvals.add(v)
            else:
                # whatever it is, we'll take it
                uvals = udict.get(k, [])
                uvals.append(v)
            udict[k] = uvals
    return {
        # set not good for JSON, have plain list
        k: list(v) if len(v) > 1 else list(v)[0]
        for k, v in udict.items()
        # do not accumulate cruft
        if len(v)}


def _meta2autofield_dict(meta, val2str=True, schema=None, consider_ucn=True):
    """Takes care of dtype conversion into unicode, potential key mappings
    and concatenation of sequence-type fields into CSV strings

    - if `consider_ucn` (default) it would copy keys from
      datalad_unique_content_properties into `meta` for that extractor
    - ... TODO ...
    """
    if consider_ucn:
        # loop over all metadata sources and the report of their unique values
        ucnprops = meta.get("datalad_unique_content_properties", {})
        for src, umeta in ucnprops.items():
            srcmeta = meta.get(src, {})
            for uk in umeta:
                if uk in srcmeta:
                    # we have a real entry for this key in the dataset metadata
                    # ignore any generated unique value list in favor of the
                    # tailored data
                    continue
                srcmeta[uk] = _listdict2dictlist(umeta[uk], strict=False) if umeta[uk] is not None else None
            if src not in meta and srcmeta:
                meta[src] = srcmeta  # assign the new one back

    srcmeta = None   # for paranoids to avoid some kind of manipulation of the last

    def _deep_kv(basekey, dct):
        """Return key/value pairs of any depth following a rule for key
        composition

        dct must be a dict
        """
        for k, v in dct.items():
            if (k != '@id' and k.startswith('@')) or k == 'datalad_unique_content_properties':
                # ignore all JSON-LD specials, but @id
                continue
            # TODO `k` might need remapping, if another key was already found
            # with the same definition
            key = u'{}{}'.format(
                basekey,
                # replace now special chars, and avoid spaces
                # `os.sep` needs to go, because whoosh uses the field name for
                # temp files during index staging, and trips over absent "directories"
                # TODO maybe even kill parentheses
                # TODO actually, it might be better to have an explicit whitelist
                k.lstrip('@').replace(os.sep, '_').replace(' ', '_').replace('-', '_').replace('.', '_').replace(':', '-')
            )
            if isinstance(v, list):
                v = _listdict2dictlist(v)

            if isinstance(v, dict):
                # dive
                for i in _deep_kv('{}.'.format(key), v):
                    yield i
            else:
                yield key, v

    def get_indexer_for(metadata_format_name: str) -> callable:
        indexer = (tuple(iter_entry_points('datalad.metadata.indexers', metadata_format_name)) or (None,))[0]
        if isinstance(indexer, EntryPoint):
            try:
                indexer_object = indexer.load()(metadata_format_name)
                return indexer_object.create_index
            except Exception as e:
                lgr.warning('Failed to load indexer %s: %s', indexer.name, exc_str(e))
        return lambda metadata: _deep_kv('', metadata)

    def _iterable_to_unicode_string(value, perform_operation):
        if perform_operation:
            if isinstance(value, (list, tuple)):
                return u' '.join(_any2unicode(i) for i in value)
            return _any2unicode(value)
        return value

    meta = meta or {}
    return {
        # Collect all meta-items which have a non-dict value type and where
        # the key not absent in a given schema.
        **{
            key: _iterable_to_unicode_string(value, val2str)
            for key, value in filter(lambda kv: not isinstance(kv[1], dict), meta.items())
            if schema is None or key in schema
        },

        # Collect all meta-items which have a dict value type and where
        # the key is neither 'datalad_unique_content_properties' nor absent
        # in a given schema.
        # These values are considered as metadata, the keys are considered to
        # be the name of the extractor, i.e. the metadata_format_name, that created
        # the metadata.
        **{
            metadata_format_name + '.' + sub_key: _iterable_to_unicode_string(sub_key_value, val2str)
            for metadata_format_name, metadata_content in filter(
                lambda kv: isinstance(kv[1], dict) and kv[0] != 'datalad_unique_content_properties',
                meta.items()
            )
            for sub_key, sub_key_value in get_indexer_for(metadata_format_name)(metadata_content)
            if schema is None or metadata_format_name + '.' + sub_key in schema
        }
    }


def _search_from_virgin_install(dataset, query):
    #
    # this is to be nice to newbies
    #
    exc_info = sys.exc_info()
    if dataset is None:
        if not ui.is_interactive:
            raise NoDatasetFound(
                "No DataLad dataset found. Specify a dataset to be "
                "searched, or run interactively to get assistance "
                "installing a queriable superdataset."
            )
        # none was provided so we could ask user whether he possibly wants
        # to install our beautiful mega-duper-super-dataset?
        # TODO: following logic could possibly benefit other actions.
        DEFAULT_DATASET_PATH = cfg.obtain('datalad.locations.default-dataset')
        if os.path.exists(DEFAULT_DATASET_PATH):
            default_ds = Dataset(DEFAULT_DATASET_PATH)
            if default_ds.is_installed():
                if ui.yesno(
                    title="No DataLad dataset found at current location",
                    text="Would you like to search the DataLad "
                         "superdataset at %r?"
                          % DEFAULT_DATASET_PATH):
                    pass
                else:
                    raise exc_info[1]
            else:
                raise NoDatasetFound(
                    "No DataLad dataset found at current location. "
                    "The DataLad superdataset location %r exists, "
                    "but does not contain an dataset."
                    % DEFAULT_DATASET_PATH)
        elif ui.yesno(
                title="No DataLad dataset found at current location",
                text="Would you like to install the DataLad "
                     "superdataset at %r?"
                     % DEFAULT_DATASET_PATH):
            from datalad.api import install
            default_ds = install(DEFAULT_DATASET_PATH, source='///')
            ui.message(
                "From now on you can refer to this dataset using the "
                "label '///'"
            )
        else:
            raise exc_info[1]

        lgr.info(
            "Performing search using DataLad superdataset %r",
            default_ds.path
        )
        for res in default_ds.search(query):
            yield res
        return
    else:
        raise  # this function is called within exception handling block


class _Search(object):
    def __init__(self, ds, **kwargs):
        self.ds = ds
        self.documenttype = self.ds.config.obtain(
            'datalad.search.index-{}-documenttype'.format(self._mode_label),
            default=self._default_documenttype)

    def __call__(self, query, max_nresults=None):
        raise NotImplementedError

    @classmethod
    def _key_matches(cls, k, regexes):
        """Return which regex the key matches
        """
        for regex in regexes:
            if re.search(regex, k):
                return regex

    def show_keys(self, *args, **kwargs):
        raise NotImplementedError(args)

    def get_query(self, query):
        """Prepare query structure specific for a search backend.

        It can also memorize within instance some parameters of the last query
        which could be used to assist output formatting/structuring later on
        """
        raise NotImplementedError

    def get_nohits_msg(self):
        """Given what it knows, provide recommendation in the case of no hits"""
        return "No search hits, wrong query? " \
               "See 'datalad search --show-keys name' for known keys " \
               "and 'datalad search --help' on how to prepare your query."


class _WhooshSearch(_Search):
    def __init__(self, ds, force_reindex=False, **kwargs):
        super(_WhooshSearch, self).__init__(ds, **kwargs)

        self.idx_obj = None
        # where does the bunny have the eggs?

        self.index_dir = opj(str(self.ds.repo.dot_git), SEARCH_INDEX_DOTGITDIR)
        self._mk_search_index(force_reindex)

    def show_keys(self, mode, regexes=None):
        """

        Parameters
        ----------
        mode: {"name"}
        regexes: list of regex
          Which keys to bother working on
        """
        if mode != 'name':
            raise NotImplementedError(
                "ATM %s can show only names, so please use show_keys with 'name'"
                % self.__class__.__name__
            )
        for k in self.idx_obj.schema.names():
            if regexes and not self._key_matches(k, regexes):
                continue
            print(u'{}'.format(k))

    def get_query(self, query):
        # parse the query string
        self._mk_parser()
        # for convenience we accept any number of args-words from the
        # shell and put them together to a single string here
        querystr = ' '.join(ensure_list(query))
        # this gives a formal whoosh query
        wquery = self.parser.parse(querystr)
        return wquery

    def _meta2doc(self, meta, val2str=True, schema=None):
        raise NotImplementedError

    def _mk_schema(self):
        raise NotImplementedError

    def _mk_parser(self):
        raise NotImplementedError

    def _mk_search_index(self, force_reindex):
        """Generic entrypoint to index generation

        The actual work that determines the structure and content of the index
        is done by functions that are passed in as arguments

        `meta2doc` - must return dict for index document from result input
        """
        from whoosh import index as widx
        from .metadata import get_ds_aggregate_db_locations
        dbloc, db_base_path = get_ds_aggregate_db_locations(self.ds)
        # what is the lastest state of aggregated metadata
        metadata_state = self.ds.repo.get_last_commit_hexsha(relpath(dbloc, start=self.ds.path))
        # use location common to all index types, they would all invalidate
        # simultaneously
        stamp_fname = opj(self.index_dir, 'datalad_metadata_state')
        index_dir = opj(self.index_dir, self._mode_label)

        if (not force_reindex) and \
                exists(index_dir) and \
                exists(stamp_fname) and \
                open(stamp_fname).read() == metadata_state:
            try:
                # TODO check that the index schema is the same
                # as the one we would have used for reindexing
                # TODO support incremental re-indexing, whoosh can do it
                idx = widx.open_dir(index_dir)
                lgr.debug(
                    'Search index contains %i documents',
                    idx.doc_count())
                self.idx_obj = idx
                return
            except widx.LockError as e:
                raise e
            except widx.IndexError as e:
                # Generic index error.
                # we try to regenerate
                lgr.warning(
                    "Cannot open existing index %s (%s), will regenerate",
                    index_dir, exc_str(e)
                )
            except widx.IndexVersionError as e:  # (msg, version, release=None)
                # Raised when you try to open an index using a format that the
                # current version of Whoosh cannot read. That is, when the index
                # you're trying to open is either not backward or forward
                # compatible with this version of Whoosh.
                # we try to regenerate
                lgr.warning(exc_str(e))
                pass
            except widx.OutOfDateError as e:
                # Raised when you try to commit changes to an index which is not
                # the latest generation.
                # this should not happen here, but if it does ... KABOOM
                raise
            except widx.EmptyIndexError as e:
                # Raised when you try to work with an index that has no indexed
                # terms.
                # we can just continue with generating an index
                pass
            except ValueError as e:
                if 'unsupported pickle protocol' in str(e):
                    lgr.warning(
                        "Cannot open existing index %s (%s), will regenerate",
                        index_dir, exc_str(e)
                    )
                else:
                    raise

        lgr.info('{} search index'.format(
            'Rebuilding' if exists(index_dir) else 'Building'))

        if not exists(index_dir):
            os.makedirs(index_dir)

        # this is a pretty cheap call that just pull this info from a file
        dsinfo = self.ds.metadata(
            get_aggregates=True,
            return_type='list',
            result_renderer='disabled')

        self._mk_schema(dsinfo)

        idx_obj = widx.create_in(index_dir, self.schema)
        idx = idx_obj.writer(
            # cache size per process
            limitmb=cfg.obtain('datalad.search.indexercachesize'),
            # disable parallel indexing for now till #1927 is resolved
            ## number of processes for indexing
            #procs=multiprocessing.cpu_count(),
            ## write separate index segments in each process for speed
            ## asks for writer.commit(optimize=True)
            #multisegment=True,
        )

        # load metadata of the base dataset and what it knows about all its subdatasets
        # (recursively)
        old_idx_size = 0
        old_ds_rpath = ''
        idx_size = 0
        log_progress(
            lgr.info,
            'autofieldidxbuild',
            'Start building search index',
            total=len(dsinfo),
            label='Building search index',
            unit=' Datasets',
        )
        for res in query_aggregated_metadata(
                reporton=self.documenttype,
                ds=self.ds,
                aps=[dict(path=self.ds.path, type='dataset')],
                # MIH: I cannot see a case when we would not want recursion (within
                # the metadata)
                recursive=True):
            # this assumes that files are reported after each dataset report,
            # and after a subsequent dataset report no files for the previous
            # dataset will be reported again
            meta = res.get('metadata', {})
            doc = self._meta2doc(meta)
            admin = {
                'type': res['type'],
                'path': relpath(res['path'], start=self.ds.path),
            }
            if 'parentds' in res:
                admin['parentds'] = relpath(res['parentds'], start=self.ds.path)
            if admin['type'] == 'dataset':
                if old_ds_rpath:
                    lgr.debug(
                        'Added %s on dataset %s',
                        single_or_plural(
                            'document',
                            'documents',
                            idx_size - old_idx_size,
                            include_count=True),
                        old_ds_rpath)
                log_progress(lgr.info, 'autofieldidxbuild',
                             'Indexed dataset at %s', old_ds_rpath,
                             update=1, increment=True)
                old_idx_size = idx_size
                old_ds_rpath = admin['path']
                admin['id'] = res.get('dsid', None)

            doc.update({k: ensure_unicode(v) for k, v in admin.items()})
            lgr.debug("Adding document to search index: {}".format(doc))
            # inject into index
            idx.add_document(**doc)
            idx_size += 1

        if old_ds_rpath:
            lgr.debug(
                'Added %s on dataset %s',
                single_or_plural(
                    'document',
                    'documents',
                    idx_size - old_idx_size,
                    include_count=True),
                old_ds_rpath)

        lgr.debug("Committing index")
        idx.commit(optimize=True)
        log_progress(
            lgr.info, 'autofieldidxbuild', 'Done building search index')

        # "timestamp" the search index to allow for automatic invalidation
        with open(stamp_fname, 'w') as f:
            f.write(metadata_state)

        lgr.info('Search index contains %i documents', idx_size)
        self.idx_obj = idx_obj

    def __call__(self, query, max_nresults=None, force_reindex=False, full_record=False):
        if max_nresults is None:
            # mode default
            max_nresults = 20

        with self.idx_obj.searcher() as searcher:
            wquery = self.get_query(query)

            # perform the actual search
            hits = searcher.search(
                wquery,
                terms=True,
                limit=max_nresults if max_nresults > 0 else None)
            # report query stats
            topstr = '{} top {}'.format(
                max_nresults,
                single_or_plural('match', 'matches', max_nresults)
            )
            lgr.info('Query completed in {} sec.{}'.format(
                hits.runtime,
                ' Reporting {}.'.format(
                    ('up to ' + topstr)
                    if max_nresults > 0
                    else 'all matches'
                )
                if not hits.is_empty()
                else ' No matches.'
            ))

            if not hits:
                return

            nhits = 0
            annotated_hits = []
            # annotate hits for full metadata report
            for i, hit in enumerate(hits):
                annotated_hit = dict(
                    path=normpath(opj(self.ds.path, hit['path'])),
                    query_matched={ensure_unicode(k): ensure_unicode(v)
                                   if isinstance(v, unicode_srctypes) else v
                                   for k, v in hit.matched_terms()},
                    parentds=normpath(
                        opj(self.ds.path, hit['parentds'])) if 'parentds' in hit else None,
                    type=hit.get('type', None))
                if not full_record:
                    annotated_hit.update(
                        refds=self.ds.path,
                        action='search',
                        status='ok',
                        logger=lgr,
                    )
                    yield annotated_hit
                else:
                    annotated_hits.append(annotated_hit)
                nhits += 1
            if full_record:
                for res in query_aggregated_metadata(
                        # type is taken from hit record
                        reporton=None,
                        ds=self.ds,
                        aps=annotated_hits,
                        # never recursive, we have direct hits already
                        recursive=False):
                    res.update(
                        refds=self.ds.path,
                        action='search',
                        status='ok',
                        logger=lgr,
                    )
                    yield res

            if max_nresults and nhits == max_nresults:
                lgr.info(
                    "Reached the limit of {}, there could be more which "
                    "were not reported.".format(topstr)
                )


class _BlobSearch(_WhooshSearch):
    _mode_label = 'textblob'
    _default_documenttype = 'datasets'

    def _meta2doc(self, meta):
        # coerce the entire flattened metadata dict into a comma-separated string
        # that also includes the keys
        return dict(meta=u', '.join(
            u'{}: {}'.format(k, v)
            for k, v in _meta2autofield_dict(
                meta,
                val2str=True,
                schema=None).items()))

    def _mk_schema(self, dsinfo):
        from whoosh import fields as wf
        from whoosh.analysis import StandardAnalyzer

        # TODO support some customizable mapping to homogenize some metadata fields
        # onto a given set of index keys
        self.schema = wf.Schema(
            id=wf.ID,
            path=wf.ID(stored=True),
            type=wf.ID(stored=True),
            parentds=wf.ID(stored=True),
            meta=wf.TEXT(
                stored=False,
                analyzer=StandardAnalyzer(minsize=2))
        )

    def _mk_parser(self):
        from whoosh import qparser as qparse

        # use whoosh default query parser for now
        parser = qparse.QueryParser(
            "meta",
            schema=self.idx_obj.schema
        )
        parser.add_plugin(qparse.FuzzyTermPlugin())
        parser.remove_plugin_class(qparse.PhrasePlugin)
        parser.add_plugin(qparse.SequencePlugin())
        self.parser = parser


class _AutofieldSearch(_WhooshSearch):
    _mode_label = 'autofield'
    _default_documenttype = 'all'

    def _meta2doc(self, meta):
        return _meta2autofield_dict(meta, val2str=True, schema=self.schema)

    def _mk_schema(self, dsinfo):
        from whoosh import fields as wf
        from whoosh.analysis import SimpleAnalyzer

        # haven for terms that have been found to be undefined
        # (for faster decision-making upon next encounter)
        # this will harvest all discovered term definitions
        definitions = {
            '@id': 'unique identifier of an entity',
            # TODO make proper JSON-LD definition
            'path': 'path name of an entity relative to the searched base dataset',
            # TODO make proper JSON-LD definition
            'parentds': 'path of the datasets that contains an entity',
            # 'type' will not come from a metadata field, hence will not be detected
            'type': 'type of a record',
        }

        schema_fields = {
            n.lstrip('@'): wf.ID(stored=True, unique=n == '@id')
            for n in definitions}

        lgr.debug('Scanning for metadata keys')
        # quick 1st pass over all dataset to gather the needed schema fields
        log_progress(
            lgr.info,
            'idxschemabuild',
            'Start building search schema',
            total=len(dsinfo),
            label='Building search schema',
            unit=' Datasets',
        )
        for res in query_aggregated_metadata(
                # XXX TODO After #2156 datasets may not necessarily carry all
                # keys in the "unique" summary
                reporton='datasets',
                ds=self.ds,
                aps=[dict(path=self.ds.path, type='dataset')],
                recursive=True):
            meta = res.get('metadata', {})
            # no stringification of values for speed, we do not need/use the
            # actual values at this point, only the keys
            idxd = _meta2autofield_dict(meta, val2str=False)

            for k in idxd:
                schema_fields[k] = wf.TEXT(stored=False,
                                           analyzer=SimpleAnalyzer())
            log_progress(lgr.info, 'idxschemabuild',
                         'Scanned dataset at %s', res['path'],
                         update=1, increment=True)
        log_progress(
            lgr.info, 'idxschemabuild', 'Done building search schema')

        self.schema = wf.Schema(**schema_fields)

    def _mk_parser(self):
        from whoosh import qparser as qparse

        parser = qparse.MultifieldParser(
            self.idx_obj.schema.names(),
            self.idx_obj.schema)
        # XXX: plugin is broken in Debian's whoosh 2.7.0-2, but already fixed
        # upstream
        parser.add_plugin(qparse.FuzzyTermPlugin())
        parser.add_plugin(qparse.GtLtPlugin())
        parser.add_plugin(qparse.SingleQuotePlugin())
        # replace field defintion to allow for colons to be part of a field's name:
        parser.replace_plugin(qparse.FieldsPlugin(expr=r"(?P<text>[()<>.\w]+|[*]):"))
        self.parser = parser


class _EGrepCSSearch(_Search):
    _mode_label = 'egrepcs'
    _default_documenttype = 'datasets'

    def __init__(self, ds, **kwargs):
        super(_EGrepCSSearch, self).__init__(ds, **kwargs)
        self._queried_keys = None  # to be memoized by get_query

    # If there were custom "per-search engine" options, we could expose
    # --consider_ucn - search through unique content properties of the dataset
    #    which might be more computationally demanding
    def __call__(self, query, max_nresults=None, consider_ucn=False, full_record=True):
        if max_nresults is None:
            # no limit by default
            max_nresults = 0
        query = self.get_query(query)

        nhits = 0
        for res in query_aggregated_metadata(
                reporton=self.documenttype,
                ds=self.ds,
                aps=[dict(path=self.ds.path, type='dataset')],
                # MIH: I cannot see a case when we would not want recursion (within
                # the metadata)
                recursive=True):
            # this assumes that files are reported after each dataset report,
            # and after a subsequent dataset report no files for the previous
            # dataset will be reported again
            meta = res.get('metadata', {})
            # produce a flattened metadata dict to search through
            doc = _meta2autofield_dict(meta, val2str=True, consider_ucn=consider_ucn)
            # inject a few basic properties into the dict
            # analog to what the other modes do in their index
            doc.update({
                k: res[k] for k in ('@id', 'type', 'path', 'parentds')
                if k in res})
            # use search instead of match to not just get hits at the start of the string
            # this will be slower, but avoids having to use actual regex syntax at the user
            # side even for simple queries
            # DOTALL is needed to handle multiline description fields and such, and still
            # be able to match content coming for a later field
            lgr.log(7, "Querying %s among %d items", query, len(doc))
            t0 = time()
            matches = {(q['query'] if isinstance(q, dict) else q, k):
                       q['query'].search(v) if isinstance(q, dict) else q.search(v)
                       for k, v in doc.items()
                       for q in query
                       if not isinstance(q, dict) or q['field'].match(k)}
            dt = time() - t0
            lgr.log(7, "Finished querying in %f sec", dt)
            # retain what actually matched
            matched = {k[1]: match.group() for k, match in matches.items() if match}
            # implement AND behavior across query expressions, but OR behavior
            # across queries matching multiple fields for a single query expression
            # for multiple queries, this makes it consistent with a query that
            # has no field specification
            if matched and len(query) == len(set(k[0] for k in matches if matches[k])):
                hit = dict(
                    res,
                    action='search',
                    query_matched=matched,
                )
                yield hit
                nhits += 1
                if max_nresults and nhits == max_nresults:
                    # report query stats
                    topstr = '{} top {}'.format(
                        max_nresults,
                        single_or_plural('match', 'matches', max_nresults)
                    )
                    lgr.info(
                        "Reached the limit of {}, there could be more which "
                        "were not reported.".format(topstr)
                    )
                    break

    def show_keys(self, mode=None, regexes=None):
        """

        Parameters
        ----------
        mode: {"name", "short", "full"}
        regexes: list of regex
          Which keys to bother working on
        """
        maxl = 100  # approx maximal line length for unique values in mode=short
        # use a dict already, later we need to map to a definition
        # meanwhile map to the values

        keys = self._get_keys(mode)

        for k in sorted(keys):
            if regexes and not self._key_matches(k, regexes):
                continue
            if mode == 'name':
                print(k)
                continue

            # do a bit more
            stat = keys[k]
            all_uvals = uvals = sorted(stat.uvals)

            stat.uvals_str = ensure_unicode(
                "{} unique values: ".format(len(all_uvals))
            )

            if mode == 'short':
                # show only up until we fill maxl
                uvals_str = ''
                uvals = []
                for v in all_uvals:
                    appendix = ('; ' if uvals else '') + v
                    if len(uvals_str) + len(appendix) > maxl - len(stat.uvals_str):
                        break
                    uvals.append(v)
                    uvals_str += appendix
            elif mode == 'full':
                pass
            else:
                raise ValueError(
                    "Unknown value for stats. Know full and short")

            stat.uvals_str += '; '.join(uvals)

            if len(all_uvals) > len(uvals):
                stat.uvals_str += \
                    '; +%s' % single_or_plural("value", "values", len(all_uvals) - len(uvals), True)

            print(
                u'{k}\n in  {stat.ndatasets} datasets\n has {stat.uvals_str}'.format(
                k=k, stat=stat
            ))
        # After #2156 datasets may not necessarily carry all
        # keys in the "unique" summary
        lgr.warning('In this search mode, the reported list of metadata keys may be incomplete')


    def _get_keys(self, mode=None):
        """Return keys and their statistics if mode != 'name'."""
        class key_stat:
            def __init__(self):
                self.ndatasets = 0  # how many datasets have this field
                self.uvals = set()

        from collections import defaultdict
        keys = defaultdict(key_stat)
        for res in query_aggregated_metadata(
                # XXX TODO After #2156 datasets may not necessarily carry all
                # keys in the "unique" summary
                reporton='datasets',
                ds=self.ds,
                aps=[dict(path=self.ds.path, type='dataset')],
                recursive=True):
            meta = res.get('metadata', {})
            # inject a few basic properties into the dict
            # analog to what the other modes do in their index
            meta.update({
                k: res.get(k, None) for k in ('@id', 'type', 'path', 'parentds')
                # parentds is tricky all files will have it, but the dataset
                # queried above might not (single dataset), let's force it in
                if k == 'parentds' or k in res})

            # no stringification of values for speed
            idxd = _meta2autofield_dict(meta, val2str=False)

            for k, kvals in idxd.items():
                # TODO deal with conflicting definitions when available
                keys[k].ndatasets += 1
                if mode == 'name':
                    continue
                keys[k].uvals |= self.get_repr_uvalues(kvals)
        return keys

    def get_repr_uvalues(self, kvals):
        kvals_set = set()
        if not kvals:
            return kvals_set
        kvals_iter = (
            kvals
            if hasattr(kvals, '__iter__') and not isinstance(kvals, (str, bytes))
            else [kvals]
        )
        return set(shortened_repr(x, 50) for x in kvals_iter)

    def get_query(self, query):
        query = ensure_list(query)
        simple_fieldspec = re.compile(r"(?P<field>\S*?):(?P<query>.*)")
        quoted_fieldspec = re.compile(r"'(?P<field>[^']+?)':(?P<query>.*)")
        query_rec_matches = [
            simple_fieldspec.match(q) or
            quoted_fieldspec.match(q) or
            q
            for q in query]
        query_group_dicts_only = [
            q.groupdict() for q in query_rec_matches if hasattr(q, 'groupdict')
        ]
        self._queried_keys = [
            qgd['field']
            for qgd in query_group_dicts_only
            if ('field' in qgd and qgd['field'])
        ]
        if len(query_group_dicts_only) != len(query_rec_matches):
            # we had a query element without field specification add
            # None as an indicator of that
            self._queried_keys.append(None)
        # expand matches, compile expressions
        query = [
            {k: self._compile_query(v) for k, v in q.groupdict().items()}
            if hasattr(q, 'groupdict') else self._compile_query(q)
            for q in query_rec_matches
        ]

        # turn "empty" field specs into simple queries
        # this is used to forcibly disable field-based search
        # e.g. when searching for a value
        query = [q['query']
                 if isinstance(q, dict) and q['field'].pattern == '' else q
                 for q in query]
        return query

    def _xfm_query(self, q):
        # implement potential transformations of regex before they get compiled
        return q

    def _compile_query(self, q):
        """xfm and compile the query, with informative exception if query is incorrect
        """
        q_xfmed = self._xfm_query(q)
        try:
            return re.compile(q_xfmed)
        except re.error as exc:
            omsg = " (original: '%s')" % q if q != q_xfmed else ''
            raise ValueError(
                "regular expression '%s'%s is incorrect: %s"
                % (q_xfmed, omsg, exc)
            )

    def get_nohits_msg(self):
        """Given the query and performed search, provide recommendation

        Quite often a key in the query is mistyped or I simply query for something
        which is not actually known.  It requires --show-keys  run first, doing
        visual search etc to mitigate.  Here we can analyze either all queried
        keys are actually known, and if not known -- what would be the ones available.

        Returns
        -------
        str
          A sentence or a paragraph to be logged/output
        """
        #
        queried_keys = self._queried_keys[:]
        if queried_keys and None in queried_keys:
            queried_keys.pop(queried_keys.index(None))
        if not queried_keys:
            return  # No keys were queried, we are of no use here
        known_keys = self._get_keys(mode='name')
        unknown_keys = sorted(list(set(queried_keys).difference(known_keys)))
        if not unknown_keys:
            return  # again we are of no help here
        msg = super(_EGrepCSSearch, self).get_nohits_msg()
        msg += " Following keys were not found in available metadata: %s. " \
              % ", ".join(unknown_keys)
        suggestions_msg = get_suggestions_msg(unknown_keys, known_keys)
        if suggestions_msg:
            msg += ' ' + suggestions_msg
        return msg


class _EGrepSearch(_EGrepCSSearch):
    _mode_label = 'egrep'
    _default_documenttype = 'datasets'

    def _xfm_query(self, q):
        if q == q.lower():
            # we have no upper case symbol in the query, go case-insensitive
            return '(?i){}'.format(q)
        else:
            return q


@build_doc
class Search(Interface):
    """Search dataset metadata

    DataLad can search metadata extracted from a dataset and/or aggregated into
    a superdataset (see the `aggregate-metadata` command). This makes it
    possible to discover datasets, or individual files in a dataset even when
    they are not available locally.

    Ultimately DataLad metadata are a graph of linked data structures. However,
    this command does not (yet) support queries that can exploit all
    information stored in the metadata. At the moment the following search
    modes are implemented that represent different trade-offs between the
    expressiveness of a query and the computational and storage resources
    required to execute a query.

    - egrep (default)

    - egrepcs [case-sensitive egrep]

    - textblob

    - autofield

    An alternative default mode can be configured by tuning the
    configuration variable 'datalad.search.default-mode'::

      [datalad "search"]
        default-mode = egrepcs

    Each search mode has its own default configuration for what kind of
    documents to query. The respective default can be changed via configuration
    variables::

      [datalad "search"]
        index-<mode_name>-documenttype = (all|datasets|files)


    *Mode: egrep/egrepcs*

    These search modes are largely ignorant of the metadata structure, and
    simply perform matching of a search pattern against a flat
    string-representation of metadata. This is advantageous when the query is
    simple and the metadata structure is irrelevant, or precisely known.
    Moreover, it does not require a search index, hence results can be reported
    without an initial latency for building a search index when the underlying
    metadata has changed (e.g. due to a dataset update). By default, these
    search modes only consider datasets and do not investigate records for
    individual files for speed reasons. Search results are reported in the
    order in which they were discovered.

    Queries can make use of Python regular expression syntax
    (https://docs.python.org/3/library/re.html). In `egrep` mode, matching is
    case-insensitive when the query does not contain upper case characters, but
    is case-sensitive when it does. In `egrepcs` mode, matching is always
    case-sensitive. Expressions will match anywhere in a metadata string, not
    only at the start.

    When multiple queries are given, all queries have to match for a search hit
    (AND behavior).

    It is possible to search individual metadata key/value items by prefixing
    the query with a metadata key name, separated by a colon (':'). The key
    name can also be a regular expression to match multiple keys. A query match
    happens when any value of an item with a matching key name matches the query
    (OR behavior). See examples for more information.

    Examples:

      Query for (what happens to be) an author::

        % datalad search haxby

      Queries are case-INsensitive when the query contains no upper case characters,
      and can be regular expressions. Use `egrepcs` mode when it is desired
      to perform a case-sensitive lowercase match::

        % datalad search --mode egrepcs halchenko.*haxby

      This search mode performs NO analysis of the metadata content.  Therefore
      queries can easily fail to match. For example, the above query implicitly
      assumes that authors are listed in alphabetical order.  If that is the
      case (which may or may not be true), the following query would yield NO
      hits::

        % datalad search Haxby.*Halchenko

      The ``textblob`` search mode represents an alternative that is more
      robust in such cases.

      For more complex queries multiple query expressions can be provided that
      all have to match to be considered a hit (AND behavior). This query
      discovers all files (non-default behavior) that match 'bids.type=T1w'
      AND 'nifti1.qform_code=scanner'::

        % datalad -c datalad.search.index-egrep-documenttype=all search bids.type:T1w nifti1.qform_code:scanner

      Key name selectors can also be expressions, which can be used to select
      multiple keys or construct "fuzzy" queries. In such cases a query matches
      when any item with a matching key matches the query (OR behavior).
      However, multiple queries are always evaluated using an AND conjunction.
      The following query extends the example above to match any files that
      have either 'nifti1.qform_code=scanner' or 'nifti1.sform_code=scanner'::

        % datalad -c datalad.search.index-egrep-documenttype=all search bids.type:T1w nifti1.(q|s)form_code:scanner

    *Mode: textblob*

    This search mode is very similar to the ``egrep`` mode, but with a few key
    differences. A search index is built from the string-representation of
    metadata records. By default, only datasets are included in this index, hence
    the indexing is usually completed within a few seconds, even for hundreds
    of datasets. This mode uses its own query language (not regular expressions)
    that is similar to other search engines. It supports logical conjunctions
    and fuzzy search terms. More information on this is available from the Whoosh
    project (search engine implementation):

      - Description of the Whoosh query language:
        http://whoosh.readthedocs.io/en/latest/querylang.html)

      - Description of a number of query language customizations that are
        enabled in DataLad, such as, fuzzy term matching:
        http://whoosh.readthedocs.io/en/latest/parsing.html#common-customizations

    Importantly, search hits are scored and reported in order of descending
    relevance, hence limiting the number of search results is more meaningful
    than in the 'egrep' mode and can also reduce the query duration.

    Examples:

      Search for (what happens to be) two authors, regardless of the order in
      which those names appear in the metadata::

        % datalad search --mode textblob halchenko haxby

      Fuzzy search when you only have an approximate idea what you are looking
      for or how it is spelled::

        % datalad search --mode textblob haxbi~

      Very fuzzy search, when you are basically only confident about the first
      two characters and how it sounds approximately (or more precisely: allow
      for three edits and require matching of the first two characters)::

        % datalad search --mode textblob haksbi~3/2

      Combine fuzzy search with logical constructs::

        % datalad search --mode textblob 'haxbi~ AND (hanke OR halchenko)'


    *Mode: autofield*

    This mode is similar to the 'textblob' mode, but builds a vastly more
    detailed search index that represents individual metadata variables as
    individual fields. By default, this search index includes records for
    datasets and individual fields, hence it can grow very quickly into
    a huge structure that can easily take an hour or more to build and require
    more than a GB of storage. However, limiting it to documents on datasets
    (see above) retains the enhanced expressiveness of queries while
    dramatically reducing the resource demands.

    Examples:

      List names of search index fields (auto-discovered from the set of
      indexed datasets) which either have a field starting with "age" or
      "gender"::

        % datalad search --mode autofield --show-keys name '\.age' '\.gender'

      Fuzzy search for datasets with an author that is specified in a particular
      metadata field::

        % datalad search --mode autofield bids.author:haxbi~ type:dataset

      Search for individual files that carry a particular description
      prefix in their 'nifti1' metadata::

        % datalad search --mode autofield nifti1.description:FSL* type:file


    *Reporting*

    Search hits are returned as standard DataLad results. On the command line
    the '--output-format' (or '-f') option can be used to tweak results for
    further processing.

    Examples:

      Format search hits as a JSON stream (one hit per line)::

        % datalad -f json search haxby

      Custom formatting: which terms matched the query of particular
      results. Useful for investigating fuzzy search results::

        $ datalad -f '{path}: {query_matched}' search --mode autofield bids.author:haxbi~
    """
    _params_ = dict(
        dataset=Parameter(
            args=("-d", "--dataset"),
            doc="""specify the dataset to perform the query operation on. If
            no dataset is given, an attempt is made to identify the dataset
            based on the current working directory and/or the `path` given""",
            constraints=EnsureDataset() | EnsureNone()),
        query=Parameter(
            args=("query",),
            metavar='QUERY',
            nargs="*",
            doc="""query string, supported syntax and features depends on the
            selected search mode (see documentation)"""),
        force_reindex=Parameter(
            args=("--reindex",),
            dest='force_reindex',
            action='store_true',
            doc="""force rebuilding the search index, even if no change in the
            dataset's state has been detected, for example, when the index
            documenttype configuration has changed."""),
        max_nresults=Parameter(
            args=("--max-nresults",),
            doc="""maxmimum number of search results to report. Setting this
            to 0 will report all search matches. Depending on the mode this
            can search substantially slower. If not specified, a
            mode-specific default setting will be used.""",
            constraints=EnsureInt() | EnsureNone()),
        mode=Parameter(
            args=("--mode",),
            choices=('egrep', 'textblob', 'autofield'),
            doc="""Mode of search index structure and content. See section
            SEARCH MODES for details."""),
        full_record=Parameter(
            args=("--full-record", '-f'),
            action='store_true',
            doc="""If set, return the full metadata record for each search hit.
            Depending on the search mode this might require additional queries.
            By default, only data that is available to the respective search modes
            is returned. This always includes essential information, such as the
            path and the type."""),
        show_keys=Parameter(
            args=('--show-keys',),
            choices=('name', 'short', 'full'),
            default=None,
            doc="""if given, a list of known search keys is shown. If 'name' -
            only the name is printed one per line. If 'short' or 'full',
            statistics (in how many datasets, and how many unique values) are
            printed. 'short' truncates the listing of unique values.
            QUERY, if provided, is regular expressions any of which keys should
            contain.
            No other action is performed (except for reindexing), even if other
            arguments are given. Each key is accompanied by a term definition in
            parenthesis (TODO). In most cases a definition is given in the form
            of a URL. If an ontology definition for a term is known, this URL
            can resolve to a webpage that provides a comprehensive definition
            of the term. However, for speed reasons term resolution is solely done
            on information contained in a local dataset's metadata, and definition
            URLs might be outdated or point to no longer existing resources."""),
        show_query=Parameter(
            args=('--show-query',),
            action='store_true',
            doc="""if given, the formal query that was generated from the given
            query string is shown, but not actually executed. This is mostly useful
            for debugging purposes."""),
    )

    @staticmethod
    @datasetmethod(name='search')
    @eval_results
    def __call__(query=None,
                 dataset=None,
                 force_reindex=False,
                 max_nresults=None,
                 mode=None,
                 full_record=False,
                 show_keys=None,
                 show_query=False):
        try:
            ds = require_dataset(dataset, check_installed=True, purpose='dataset search')
            if ds.id is None:
                raise NoDatasetFound(
                    "This does not seem to be a dataset (no DataLad dataset ID "
                    "found). 'datalad create --force %s' can initialize "
                    "this repository as a DataLad dataset" % ds.path)
        except NoDatasetFound:
            for r in _search_from_virgin_install(dataset, query):
                yield r
            return

        if mode is None:
            # let's get inspired by what the dataset/user think is
            # default
            mode = ds.config.obtain('datalad.search.default-mode')

        if mode == 'egrep':
            searcher = _EGrepSearch
        elif mode == 'egrepcs':
            searcher = _EGrepCSSearch
        elif mode == 'textblob':
            searcher = _BlobSearch
        elif mode == 'autofield':
            searcher = _AutofieldSearch
        else:
            raise ValueError(
                'unknown search mode "{}"'.format(mode))

        searcher = searcher(ds, force_reindex=force_reindex)

        if show_keys:
            searcher.show_keys(show_keys, regexes=query)
            return

        if not query:
            return

        if show_query:
            print(repr(searcher.get_query(query)))
            return

        nhits = 0
        for r in searcher(
                query,
                max_nresults=max_nresults,
                full_record=full_record):
            nhits += 1
            yield r
        if not nhits:
            lgr.info(searcher.get_nohits_msg() or 'no hits')

```
--- END FILE ---

Please first localize the bug based on the issue statement, and then generate *SEARCH/REPLACE* edits to fix the issue.

Every *SEARCH/REPLACE* edit must use this format:
1. The file path
2. The start of search block: <<<<<<< SEARCH
3. A contiguous chunk of lines to search for in the existing source code
4. The dividing line: =======
5. The lines to replace into the source code
6. The end of the replace block: >>>>>>> REPLACE

Here is an example:

```python
### mathweb/flask/app.py
<<<<<<< SEARCH
from flask import Flask
=======
import math
from flask import Flask
>>>>>>> REPLACE
```
</solution>

Please note that the *SEARCH/REPLACE* edit REQUIRES PROPER INDENTATION. If you would like to add the line '        print(x)', you must fully write that out, with all those spaces before the code!
Wrap each *SEARCH/REPLACE* edit in a code block as shown in the example above. If you have multiple *SEARCH/REPLACE* edits, use a separate code block for each one.
user