{
    "file_diffs": [
        {
            "old_file_content": "",
            "new_file_content": "### 🐛 Bug Fixes\n\n- Interface-specific (python vs CLI) doc generation for commands and their parameters was broken when brackets were used within the interface markups. \n  Fixes [#7225](https://github.com/datalad/datalad/issues/7225) via\n  [PR #7226](https://github.com/datalad/datalad/pull/7226)\n  (by [@bpoldrack](https://github.com/bpoldrack))\n",
            "header": {
                "file": {
                    "path": "changelog.d/pr-7226.md"
                },
                "misc_line": "new file mode 100644"
            },
            "index_line": {
                "old_commit_hash": "000000000",
                "new_commit_hash": "03f0d2171",
                "mode": ""
            },
            "is_binary_file": false,
            "binary_line": null,
            "minus_file": {
                "path": "/dev/null"
            },
            "plus_file": {
                "path": "b/changelog.d/pr-7226.md"
            },
            "hunks": [
                {
                    "descriptor": {
                        "old_range": {
                            "start": 0,
                            "length": 0
                        },
                        "new_range": {
                            "start": 1,
                            "length": 6
                        },
                        "section": ""
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "### 🐛 Bug Fixes",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "added"
                            },
                            {
                                "content": "- Interface-specific (python vs CLI) doc generation for commands and their parameters was broken when brackets were used within the interface markups. ",
                                "type": "added"
                            },
                            {
                                "content": "  Fixes [#7225](https://github.com/datalad/datalad/issues/7225) via",
                                "type": "added"
                            },
                            {
                                "content": "  [PR #7226](https://github.com/datalad/datalad/pull/7226)",
                                "type": "added"
                            },
                            {
                                "content": "  (by [@bpoldrack](https://github.com/bpoldrack))",
                                "type": "added"
                            }
                        ]
                    },
                    "modified_entities": [],
                    "added_entities": [],
                    "deleted_entities": []
                }
            ]
        },
        {
            "old_file_content": "\"\"\"Utilities and definitions for DataLad command interfaces\"\"\"\n\n# TODO this should be a dochelper\nfrom datalad.interface.base import dedent_docstring\n\n# Some known extensions and their commands to suggest whenever lookup fails\n_known_extension_commands = {\n    'datalad-container': (\n        'containers-list', 'containers-remove', 'containers-add',\n        'containers-run'),\n    'datalad-crawler': ('crawl', 'crawl-init'),\n    'datalad-deprecated': ('ls',),\n    'datalad-neuroimaging': ('bids2scidata',)\n}\n\n_deprecated_commands = {\n    'add': \"save\",\n    'uninstall': 'drop',\n}\n\n\ndef get_cmd_ex(interface):\n    \"\"\"Return the examples for the command defined by 'interface'.\n\n    Parameters\n    ----------\n    interface : subclass of Interface\n    \"\"\"\n    from datalad.interface.base import build_example\n    intf_ex = \"\\n\\n*Examples*\\n\\n\"\n    for example in interface._examples_:\n        intf_ex += build_example(example, api='cmdline')\n    return intf_ex\n\n\ndef get_cmdline_command_name(intfspec):\n    \"\"\"Given an interface specification return a cmdline command name\"\"\"\n    if len(intfspec) > 2:\n        name = intfspec[2]\n    else:\n        name = intfspec[0].split('.')[-1].replace('_', '-')\n    return name\n\n\ndef alter_interface_docs_for_cmdline(docs):\n    \"\"\"Apply modifications to interface docstrings for cmdline doc use.\"\"\"\n    # central place to alter the impression of docstrings,\n    # like removing Python API specific sections, and argument markup\n    if not docs:\n        return docs\n    import re\n    import textwrap\n\n    docs = dedent_docstring(docs)\n    # clean cmdline sections\n    docs = re.sub(\n        r'\\|\\| PYTHON \\>\\>.*?\\<\\< PYTHON \\|\\|',\n        '',\n        docs,\n        flags=re.MULTILINE | re.DOTALL)\n    # clean cmdline in-line bits\n    docs = re.sub(\n        r'\\[PY:\\s[^\\[\\]]*\\sPY\\]',\n        '',\n        docs,\n        flags=re.MULTILINE | re.DOTALL)\n    docs = re.sub(\n        r'\\[CMD:\\s([^\\[\\]]*)\\sCMD\\]',\n        lambda match: match.group(1),\n        docs,\n        flags=re.MULTILINE)\n    docs = re.sub(\n        r'\\|\\| CMDLINE \\>\\>(.*?)\\<\\< CMDLINE \\|\\|',\n        lambda match: match.group(1),\n        docs,\n        flags=re.MULTILINE | re.DOTALL)\n    # remove :role:`...` RST markup for cmdline docs\n    docs = re.sub(\n        r':\\S+:`[^`]*`[\\\\]*',\n        lambda match: ':'.join(match.group(0).split(':')[2:]).strip('`\\\\'),\n        docs,\n        flags=re.MULTILINE | re.DOTALL)\n    # make the handbook doc references more accessible\n    # the URL is a redirect configured at readthedocs\n    docs = re.sub(\n        r'(handbook:[0-9]-[0-9]*)',\n        '\\\\1 (http://handbook.datalad.org/symbols)',\n        docs)\n    # remove None constraint. In general, `None` on the cmdline means don't\n    # give option at all, but specifying `None` explicitly is practically\n    # impossible\n    docs = re.sub(\n        r',\\sor\\svalue\\smust\\sbe\\s`None`',\n        '',\n        docs,\n        flags=re.MULTILINE | re.DOTALL)\n    # capitalize variables and remove backticks to uniformize with\n    # argparse output\n    docs = re.sub(\n        r'([^`]+)`([a-zA-Z0-9_]+)`([^`]+)',\n        lambda match: f'{match.group(1)}{match.group(2).upper()}{match.group(3)}',\n        docs)\n    # select only the cmdline alternative from argument specifications\n    docs = re.sub(\n        r'``([a-zA-Z0-9_,.]+)\\|\\|([a-zA-Z0-9-,.]+)``',\n        lambda match: f'``{match.group(2)}``',\n        docs)\n    # clean up sphinx API refs\n    docs = re.sub(\n        r'\\~datalad\\.api\\.\\S*',\n        lambda match: \"`{0}`\".format(match.group(0)[13:]),\n        docs)\n    # dedicated support for version markup\n    docs = docs.replace('.. versionadded::', 'New in version')\n    docs = docs.replace('.. versionchanged::', 'Changed in version')\n    docs = docs.replace('.. deprecated::', 'Deprecated in version')\n    # Remove RST paragraph markup\n    docs = re.sub(\n        r'^.. \\S+::',\n        lambda match: match.group(0)[3:-2].upper(),\n        docs,\n        flags=re.MULTILINE)\n    docs = re.sub(\n        r'^([ ]*)\\|\\| REFLOW \\>\\>\\n(.*?)\\<\\< REFLOW \\|\\|',\n        lambda match: textwrap.fill(match.group(2), subsequent_indent=match.group(1)),\n        docs,\n        flags=re.MULTILINE | re.DOTALL)\n    return docs\n",
            "new_file_content": "\"\"\"Utilities and definitions for DataLad command interfaces\"\"\"\n\n# TODO this should be a dochelper\nfrom datalad.interface.base import dedent_docstring\n\n# Some known extensions and their commands to suggest whenever lookup fails\n_known_extension_commands = {\n    'datalad-container': (\n        'containers-list', 'containers-remove', 'containers-add',\n        'containers-run'),\n    'datalad-crawler': ('crawl', 'crawl-init'),\n    'datalad-deprecated': ('ls',),\n    'datalad-neuroimaging': ('bids2scidata',)\n}\n\n_deprecated_commands = {\n    'add': \"save\",\n    'uninstall': 'drop',\n}\n\n\ndef get_cmd_ex(interface):\n    \"\"\"Return the examples for the command defined by 'interface'.\n\n    Parameters\n    ----------\n    interface : subclass of Interface\n    \"\"\"\n    from datalad.interface.base import build_example\n    intf_ex = \"\\n\\n*Examples*\\n\\n\"\n    for example in interface._examples_:\n        intf_ex += build_example(example, api='cmdline')\n    return intf_ex\n\n\ndef get_cmdline_command_name(intfspec):\n    \"\"\"Given an interface specification return a cmdline command name\"\"\"\n    if len(intfspec) > 2:\n        name = intfspec[2]\n    else:\n        name = intfspec[0].split('.')[-1].replace('_', '-')\n    return name\n\n\ndef alter_interface_docs_for_cmdline(docs):\n    \"\"\"Apply modifications to interface docstrings for cmdline doc use.\"\"\"\n    # central place to alter the impression of docstrings,\n    # like removing Python API specific sections, and argument markup\n    if not docs:\n        return docs\n    import re\n    import textwrap\n\n    docs = dedent_docstring(docs)\n    # clean cmdline sections\n    docs = re.sub(\n        r'\\|\\| PYTHON \\>\\>.*?\\<\\< PYTHON \\|\\|',\n        '',\n        docs,\n        flags=re.MULTILINE | re.DOTALL)\n    # clean cmdline in-line bits\n    docs = re.sub(\n        r'\\[PY:\\s.*?\\sPY\\]',\n        '',\n        docs,\n        flags=re.MULTILINE | re.DOTALL)\n    docs = re.sub(\n        r'\\[CMD:\\s(.*?)\\sCMD\\]',\n        lambda match: match.group(1),\n        docs,\n        flags=re.MULTILINE | re.DOTALL)\n    docs = re.sub(\n        r'\\|\\| CMDLINE \\>\\>(.*?)\\<\\< CMDLINE \\|\\|',\n        lambda match: match.group(1),\n        docs,\n        flags=re.MULTILINE | re.DOTALL)\n    # remove :role:`...` RST markup for cmdline docs\n    docs = re.sub(\n        r':\\S+:`[^`]*`[\\\\]*',\n        lambda match: ':'.join(match.group(0).split(':')[2:]).strip('`\\\\'),\n        docs,\n        flags=re.MULTILINE | re.DOTALL)\n    # make the handbook doc references more accessible\n    # the URL is a redirect configured at readthedocs\n    docs = re.sub(\n        r'(handbook:[0-9]-[0-9]*)',\n        '\\\\1 (http://handbook.datalad.org/symbols)',\n        docs)\n    # remove None constraint. In general, `None` on the cmdline means don't\n    # give option at all, but specifying `None` explicitly is practically\n    # impossible\n    docs = re.sub(\n        r',\\sor\\svalue\\smust\\sbe\\s`None`',\n        '',\n        docs,\n        flags=re.MULTILINE | re.DOTALL)\n    # capitalize variables and remove backticks to uniformize with\n    # argparse output\n    docs = re.sub(\n        r'([^`]+)`([a-zA-Z0-9_]+)`([^`]+)',\n        lambda match: f'{match.group(1)}{match.group(2).upper()}{match.group(3)}',\n        docs)\n    # select only the cmdline alternative from argument specifications\n    docs = re.sub(\n        r'``([a-zA-Z0-9_,.]+)\\|\\|([a-zA-Z0-9-,.]+)``',\n        lambda match: f'``{match.group(2)}``',\n        docs)\n    # clean up sphinx API refs\n    docs = re.sub(\n        r'\\~datalad\\.api\\.\\S*',\n        lambda match: \"`{0}`\".format(match.group(0)[13:]),\n        docs)\n    # dedicated support for version markup\n    docs = docs.replace('.. versionadded::', 'New in version')\n    docs = docs.replace('.. versionchanged::', 'Changed in version')\n    docs = docs.replace('.. deprecated::', 'Deprecated in version')\n    # Remove RST paragraph markup\n    docs = re.sub(\n        r'^.. \\S+::',\n        lambda match: match.group(0)[3:-2].upper(),\n        docs,\n        flags=re.MULTILINE)\n    docs = re.sub(\n        r'^([ ]*)\\|\\| REFLOW \\>\\>\\n(.*?)\\<\\< REFLOW \\|\\|',\n        lambda match: textwrap.fill(match.group(2), subsequent_indent=match.group(1)),\n        docs,\n        flags=re.MULTILINE | re.DOTALL)\n    return docs\n",
            "header": {
                "file": {
                    "path": "datalad/cli/interface.py"
                },
                "misc_line": null
            },
            "index_line": {
                "old_commit_hash": "6a5d60212",
                "new_commit_hash": "bee96588a",
                "mode": "100644"
            },
            "is_binary_file": false,
            "binary_line": null,
            "minus_file": {
                "path": "a/datalad/cli/interface.py"
            },
            "plus_file": {
                "path": "b/datalad/cli/interface.py"
            },
            "hunks": [
                {
                    "descriptor": {
                        "old_range": {
                            "start": 60,
                            "length": 15
                        },
                        "new_range": {
                            "start": 60,
                            "length": 15
                        },
                        "section": "def alter_interface_docs_for_cmdline(docs):"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "        flags=re.MULTILINE | re.DOTALL)",
                                "type": "context"
                            },
                            {
                                "content": "    # clean cmdline in-line bits",
                                "type": "context"
                            },
                            {
                                "content": "    docs = re.sub(",
                                "type": "context"
                            },
                            {
                                "content": "        r'\\[PY:\\s[^\\[\\]]*\\sPY\\]',",
                                "type": "deleted"
                            },
                            {
                                "content": "        r'\\[PY:\\s.*?\\sPY\\]',",
                                "type": "added"
                            },
                            {
                                "content": "        '',",
                                "type": "context"
                            },
                            {
                                "content": "        docs,",
                                "type": "context"
                            },
                            {
                                "content": "        flags=re.MULTILINE | re.DOTALL)",
                                "type": "context"
                            },
                            {
                                "content": "    docs = re.sub(",
                                "type": "context"
                            },
                            {
                                "content": "        r'\\[CMD:\\s([^\\[\\]]*)\\sCMD\\]',",
                                "type": "deleted"
                            },
                            {
                                "content": "        r'\\[CMD:\\s(.*?)\\sCMD\\]',",
                                "type": "added"
                            },
                            {
                                "content": "        lambda match: match.group(1),",
                                "type": "context"
                            },
                            {
                                "content": "        docs,",
                                "type": "context"
                            },
                            {
                                "content": "        flags=re.MULTILINE)",
                                "type": "deleted"
                            },
                            {
                                "content": "        flags=re.MULTILINE | re.DOTALL)",
                                "type": "added"
                            },
                            {
                                "content": "    docs = re.sub(",
                                "type": "context"
                            },
                            {
                                "content": "        r'\\|\\| CMDLINE \\>\\>(.*?)\\<\\< CMDLINE \\|\\|',",
                                "type": "context"
                            },
                            {
                                "content": "        lambda match: match.group(1),",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [
                        {
                            "file_name": "datalad/cli/interface.py",
                            "type": "function",
                            "name": "alter_interface_docs_for_cmdline",
                            "content": "def alter_interface_docs_for_cmdline(docs):\n    \"\"\"Apply modifications to interface docstrings for cmdline doc use.\"\"\"\n    # central place to alter the impression of docstrings,\n    # like removing Python API specific sections, and argument markup\n    if not docs:\n        return docs\n    import re\n    import textwrap\n\n    docs = dedent_docstring(docs)\n    # clean cmdline sections\n    docs = re.sub(\n        r'\\|\\| PYTHON \\>\\>.*?\\<\\< PYTHON \\|\\|',\n        '',\n        docs,\n        flags=re.MULTILINE | re.DOTALL)\n    # clean cmdline in-line bits\n    docs = re.sub(\n        r'\\[PY:\\s[^\\[\\]]*\\sPY\\]',\n        '',\n        docs,\n        flags=re.MULTILINE | re.DOTALL)\n    docs = re.sub(\n        r'\\[CMD:\\s([^\\[\\]]*)\\sCMD\\]',\n        lambda match: match.group(1),\n        docs,\n        flags=re.MULTILINE)\n    docs = re.sub(\n        r'\\|\\| CMDLINE \\>\\>(.*?)\\<\\< CMDLINE \\|\\|',\n        lambda match: match.group(1),\n        docs,\n        flags=re.MULTILINE | re.DOTALL)\n    # remove :role:`...` RST markup for cmdline docs\n    docs = re.sub(\n        r':\\S+:`[^`]*`[\\\\]*',\n        lambda match: ':'.join(match.group(0).split(':')[2:]).strip('`\\\\'),\n        docs,\n        flags=re.MULTILINE | re.DOTALL)\n    # make the handbook doc references more accessible\n    # the URL is a redirect configured at readthedocs\n    docs = re.sub(\n        r'(handbook:[0-9]-[0-9]*)',\n        '\\\\1 (http://handbook.datalad.org/symbols)',\n        docs)\n    # remove None constraint. In general, `None` on the cmdline means don't\n    # give option at all, but specifying `None` explicitly is practically\n    # impossible\n    docs = re.sub(\n        r',\\sor\\svalue\\smust\\sbe\\s`None`',\n        '',\n        docs,\n        flags=re.MULTILINE | re.DOTALL)\n    # capitalize variables and remove backticks to uniformize with\n    # argparse output\n    docs = re.sub(\n        r'([^`]+)`([a-zA-Z0-9_]+)`([^`]+)',\n        lambda match: f'{match.group(1)}{match.group(2).upper()}{match.group(3)}',\n        docs)\n    # select only the cmdline alternative from argument specifications\n    docs = re.sub(\n        r'``([a-zA-Z0-9_,.]+)\\|\\|([a-zA-Z0-9-,.]+)``',\n        lambda match: f'``{match.group(2)}``',\n        docs)\n    # clean up sphinx API refs\n    docs = re.sub(\n        r'\\~datalad\\.api\\.\\S*',\n        lambda match: \"`{0}`\".format(match.group(0)[13:]),\n        docs)\n    # dedicated support for version markup\n    docs = docs.replace('.. versionadded::', 'New in version')\n    docs = docs.replace('.. versionchanged::', 'Changed in version')\n    docs = docs.replace('.. deprecated::', 'Deprecated in version')\n    # Remove RST paragraph markup\n    docs = re.sub(\n        r'^.. \\S+::',\n        lambda match: match.group(0)[3:-2].upper(),\n        docs,\n        flags=re.MULTILINE)\n    docs = re.sub(\n        r'^([ ]*)\\|\\| REFLOW \\>\\>\\n(.*?)\\<\\< REFLOW \\|\\|',\n        lambda match: textwrap.fill(match.group(2), subsequent_indent=match.group(1)),\n        docs,\n        flags=re.MULTILINE | re.DOTALL)\n    return docs",
                            "ast_type_str": "FunctionDef",
                            "start_lineno": 45,
                            "end_lineno": 128,
                            "parent": null
                        }
                    ],
                    "added_entities": [],
                    "deleted_entities": []
                }
            ]
        },
        {
            "old_file_content": "from datalad.interface.tests.test_docs import (\n    demo_argdoc,\n    demo_doc,\n    demo_paramdoc,\n)\nfrom datalad.tests.utils_pytest import (\n    assert_false,\n    assert_in,\n    assert_not_in,\n    eq_,\n)\n\nfrom ..interface import (\n    alter_interface_docs_for_cmdline,\n    get_cmdline_command_name,\n)\n\n\ndef test_alter_interface_docs_for_cmdline():\n    alt = alter_interface_docs_for_cmdline(demo_doc)\n    alt_l = alt.split('\\n')\n    # de-dented\n    assert_false(alt_l[0].startswith(' '))\n    assert_false(alt_l[-1].startswith(' '))\n    assert_not_in('PY', alt)\n    assert_not_in('CMD', alt)\n    assert_not_in('REFLOW', alt)\n    assert_in('a b', alt)\n    assert_in('not\\n   reflowed', alt)\n    assert_in(\"Something for the cmdline only Multiline!\", alt)\n    # args\n    altarg = alter_interface_docs_for_cmdline(demo_argdoc)\n    # RST role markup\n    eq_(alter_interface_docs_for_cmdline(':murks:`me and my buddies`'),\n        'me and my buddies')\n    # spread across lines\n    eq_(alter_interface_docs_for_cmdline(':term:`Barbara\\nStreisand`'),\n        'Barbara\\nStreisand')\n    # multiple on one line\n    eq_(alter_interface_docs_for_cmdline(\n        ':term:`one` bla bla :term:`two` bla'),\n        'one bla bla two bla')\n\n    altpd = alter_interface_docs_for_cmdline(demo_paramdoc)\n    assert_not_in('python', altpd)\n    assert_in('in between', altpd)\n    assert_in('appended', altpd)\n    assert_in('cmdline', altpd)\n\n\ndef test_name_generation():\n    eq_(\n        get_cmdline_command_name((\"some.module_something\", \"SomeClass\")),\n        \"module-something\")\n    eq_(\n        get_cmdline_command_name((\n            \"some.module_something\",\n            \"SomeClass\",\n            \"override\")),\n        \"override\")\n    eq_(\n        get_cmdline_command_name((\n            \"some.module_something\",\n            \"SomeClass\",\n            \"override\",\n            \"api_ignore\")),\n        \"override\")\n",
            "new_file_content": "from datalad.interface.tests.test_docs import (\n    demo_argdoc,\n    demo_doc,\n    demo_paramdoc,\n)\nfrom datalad.tests.utils_pytest import (\n    assert_false,\n    assert_in,\n    assert_not_in,\n    eq_,\n)\n\nfrom ..interface import (\n    alter_interface_docs_for_cmdline,\n    get_cmdline_command_name,\n)\n\n\ndef test_alter_interface_docs_for_cmdline():\n    alt = alter_interface_docs_for_cmdline(demo_doc)\n    alt_l = alt.split('\\n')\n    # de-dented\n    assert_false(alt_l[0].startswith(' '))\n    assert_false(alt_l[-1].startswith(' '))\n    assert_not_in('PY', alt)\n    assert_not_in('CMD', alt)\n    assert_not_in('REFLOW', alt)\n    assert_in('a b', alt)\n    assert_in('not\\n   reflowed', alt)\n    assert_in(\"Something for the cmdline only Multiline!\", alt)\n    assert_not_in(\"Some Python-only bits\", alt)\n    assert_not_in(\"just for Python\", alt)\n    assert_in(\"just for the command line\", alt)\n    assert_in(\"multiline cli-only with [ brackets\\n[] ]\", alt)\n    assert_not_in(\"multiline\\npython-only with [ brackets [] ]\", alt)\n\n    # args\n    altarg = alter_interface_docs_for_cmdline(demo_argdoc)\n    # RST role markup\n    eq_(alter_interface_docs_for_cmdline(':murks:`me and my buddies`'),\n        'me and my buddies')\n    # spread across lines\n    eq_(alter_interface_docs_for_cmdline(':term:`Barbara\\nStreisand`'),\n        'Barbara\\nStreisand')\n    # multiple on one line\n    eq_(alter_interface_docs_for_cmdline(\n        ':term:`one` bla bla :term:`two` bla'),\n        'one bla bla two bla')\n\n    altpd = alter_interface_docs_for_cmdline(demo_paramdoc)\n    assert_not_in(\"PY\", altpd)\n    assert_not_in(\"CMD\", altpd)\n    assert_not_in('python', altpd)\n    assert_not_in(\"python-only with [ some brackets []\", altpd)\n    assert_in('in between', altpd)\n    assert_in('appended', altpd)\n    assert_in('cmdline', altpd)\n    assert_in(\"multiline cli-only [\\n  brackets included \"\n              \"[ can we also have || ?]\", altpd)\n\n\ndef test_name_generation():\n    eq_(\n        get_cmdline_command_name((\"some.module_something\", \"SomeClass\")),\n        \"module-something\")\n    eq_(\n        get_cmdline_command_name((\n            \"some.module_something\",\n            \"SomeClass\",\n            \"override\")),\n        \"override\")\n    eq_(\n        get_cmdline_command_name((\n            \"some.module_something\",\n            \"SomeClass\",\n            \"override\",\n            \"api_ignore\")),\n        \"override\")\n",
            "header": {
                "file": {
                    "path": "datalad/cli/tests/test_interface.py"
                },
                "misc_line": null
            },
            "index_line": {
                "old_commit_hash": "7090988f6",
                "new_commit_hash": "9a00cbcb2",
                "mode": "100644"
            },
            "is_binary_file": false,
            "binary_line": null,
            "minus_file": {
                "path": "a/datalad/cli/tests/test_interface.py"
            },
            "plus_file": {
                "path": "b/datalad/cli/tests/test_interface.py"
            },
            "hunks": [
                {
                    "descriptor": {
                        "old_range": {
                            "start": 28,
                            "length": 6
                        },
                        "new_range": {
                            "start": 28,
                            "length": 12
                        },
                        "section": "def test_alter_interface_docs_for_cmdline():"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "    assert_in('a b', alt)",
                                "type": "context"
                            },
                            {
                                "content": "    assert_in('not\\n   reflowed', alt)",
                                "type": "context"
                            },
                            {
                                "content": "    assert_in(\"Something for the cmdline only Multiline!\", alt)",
                                "type": "context"
                            },
                            {
                                "content": "    assert_not_in(\"Some Python-only bits\", alt)",
                                "type": "added"
                            },
                            {
                                "content": "    assert_not_in(\"just for Python\", alt)",
                                "type": "added"
                            },
                            {
                                "content": "    assert_in(\"just for the command line\", alt)",
                                "type": "added"
                            },
                            {
                                "content": "    assert_in(\"multiline cli-only with [ brackets\\n[] ]\", alt)",
                                "type": "added"
                            },
                            {
                                "content": "    assert_not_in(\"multiline\\npython-only with [ brackets [] ]\", alt)",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "added"
                            },
                            {
                                "content": "    # args",
                                "type": "context"
                            },
                            {
                                "content": "    altarg = alter_interface_docs_for_cmdline(demo_argdoc)",
                                "type": "context"
                            },
                            {
                                "content": "    # RST role markup",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [
                        {
                            "file_name": "datalad/cli/tests/test_interface.py",
                            "type": "function",
                            "name": "test_alter_interface_docs_for_cmdline",
                            "content": "def test_alter_interface_docs_for_cmdline():\n    alt = alter_interface_docs_for_cmdline(demo_doc)\n    alt_l = alt.split('\\n')\n    # de-dented\n    assert_false(alt_l[0].startswith(' '))\n    assert_false(alt_l[-1].startswith(' '))\n    assert_not_in('PY', alt)\n    assert_not_in('CMD', alt)\n    assert_not_in('REFLOW', alt)\n    assert_in('a b', alt)\n    assert_in('not\\n   reflowed', alt)\n    assert_in(\"Something for the cmdline only Multiline!\", alt)\n    assert_not_in(\"Some Python-only bits\", alt)\n    assert_not_in(\"just for Python\", alt)\n    assert_in(\"just for the command line\", alt)\n    assert_in(\"multiline cli-only with [ brackets\\n[] ]\", alt)\n    assert_not_in(\"multiline\\npython-only with [ brackets [] ]\", alt)\n\n    # args\n    altarg = alter_interface_docs_for_cmdline(demo_argdoc)\n    # RST role markup\n    eq_(alter_interface_docs_for_cmdline(':murks:`me and my buddies`'),\n        'me and my buddies')\n    # spread across lines\n    eq_(alter_interface_docs_for_cmdline(':term:`Barbara\\nStreisand`'),\n        'Barbara\\nStreisand')\n    # multiple on one line\n    eq_(alter_interface_docs_for_cmdline(\n        ':term:`one` bla bla :term:`two` bla'),\n        'one bla bla two bla')\n\n    altpd = alter_interface_docs_for_cmdline(demo_paramdoc)\n    assert_not_in(\"PY\", altpd)\n    assert_not_in(\"CMD\", altpd)\n    assert_not_in('python', altpd)\n    assert_not_in(\"python-only with [ some brackets []\", altpd)\n    assert_in('in between', altpd)\n    assert_in('appended', altpd)\n    assert_in('cmdline', altpd)\n    assert_in(\"multiline cli-only [\\n  brackets included \"\n              \"[ can we also have || ?]\", altpd)",
                            "ast_type_str": "FunctionDef",
                            "start_lineno": 19,
                            "end_lineno": 59,
                            "parent": null
                        }
                    ],
                    "added_entities": [],
                    "deleted_entities": []
                },
                {
                    "descriptor": {
                        "old_range": {
                            "start": 42,
                            "length": 10
                        },
                        "new_range": {
                            "start": 48,
                            "length": 15
                        },
                        "section": "def test_alter_interface_docs_for_cmdline():"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "        'one bla bla two bla')",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "    altpd = alter_interface_docs_for_cmdline(demo_paramdoc)",
                                "type": "context"
                            },
                            {
                                "content": "    assert_not_in(\"PY\", altpd)",
                                "type": "added"
                            },
                            {
                                "content": "    assert_not_in(\"CMD\", altpd)",
                                "type": "added"
                            },
                            {
                                "content": "    assert_not_in('python', altpd)",
                                "type": "context"
                            },
                            {
                                "content": "    assert_not_in(\"python-only with [ some brackets []\", altpd)",
                                "type": "added"
                            },
                            {
                                "content": "    assert_in('in between', altpd)",
                                "type": "context"
                            },
                            {
                                "content": "    assert_in('appended', altpd)",
                                "type": "context"
                            },
                            {
                                "content": "    assert_in('cmdline', altpd)",
                                "type": "context"
                            },
                            {
                                "content": "    assert_in(\"multiline cli-only [\\n  brackets included \"",
                                "type": "added"
                            },
                            {
                                "content": "              \"[ can we also have || ?]\", altpd)",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "def test_name_generation():",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [
                        {
                            "file_name": "datalad/cli/tests/test_interface.py",
                            "type": "function",
                            "name": "test_alter_interface_docs_for_cmdline",
                            "content": "def test_alter_interface_docs_for_cmdline():\n    alt = alter_interface_docs_for_cmdline(demo_doc)\n    alt_l = alt.split('\\n')\n    # de-dented\n    assert_false(alt_l[0].startswith(' '))\n    assert_false(alt_l[-1].startswith(' '))\n    assert_not_in('PY', alt)\n    assert_not_in('CMD', alt)\n    assert_not_in('REFLOW', alt)\n    assert_in('a b', alt)\n    assert_in('not\\n   reflowed', alt)\n    assert_in(\"Something for the cmdline only Multiline!\", alt)\n    assert_not_in(\"Some Python-only bits\", alt)\n    assert_not_in(\"just for Python\", alt)\n    assert_in(\"just for the command line\", alt)\n    assert_in(\"multiline cli-only with [ brackets\\n[] ]\", alt)\n    assert_not_in(\"multiline\\npython-only with [ brackets [] ]\", alt)\n\n    # args\n    altarg = alter_interface_docs_for_cmdline(demo_argdoc)\n    # RST role markup\n    eq_(alter_interface_docs_for_cmdline(':murks:`me and my buddies`'),\n        'me and my buddies')\n    # spread across lines\n    eq_(alter_interface_docs_for_cmdline(':term:`Barbara\\nStreisand`'),\n        'Barbara\\nStreisand')\n    # multiple on one line\n    eq_(alter_interface_docs_for_cmdline(\n        ':term:`one` bla bla :term:`two` bla'),\n        'one bla bla two bla')\n\n    altpd = alter_interface_docs_for_cmdline(demo_paramdoc)\n    assert_not_in(\"PY\", altpd)\n    assert_not_in(\"CMD\", altpd)\n    assert_not_in('python', altpd)\n    assert_not_in(\"python-only with [ some brackets []\", altpd)\n    assert_in('in between', altpd)\n    assert_in('appended', altpd)\n    assert_in('cmdline', altpd)\n    assert_in(\"multiline cli-only [\\n  brackets included \"\n              \"[ can we also have || ?]\", altpd)",
                            "ast_type_str": "FunctionDef",
                            "start_lineno": 19,
                            "end_lineno": 59,
                            "parent": null
                        }
                    ],
                    "added_entities": [],
                    "deleted_entities": []
                }
            ]
        },
        {
            "old_file_content": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 et:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"High-level interface generation\n\n\"\"\"\n\n__docformat__ = 'restructuredtext'\n\nimport logging\n\nlgr = logging.getLogger('datalad.interface.base')\n\nimport os\nimport re\nimport textwrap\nimport warnings\nfrom abc import (\n    ABC,\n    abstractmethod,\n)\nfrom collections import OrderedDict\nfrom importlib import import_module\n\nimport datalad\nfrom datalad.distribution.dataset import (\n    Dataset,\n    resolve_path,\n)\nfrom datalad.interface.common_opts import eval_params\nfrom datalad.support.exceptions import CapturedException\n\ndefault_logchannels = {\n    '': 'debug',\n    'ok': 'debug',\n    'notneeded': 'debug',\n    'impossible': 'warning',\n    'error': 'error',\n}\n\n\ndef get_api_name(intfspec):\n    \"\"\"Given an interface specification return an API name for it\"\"\"\n    if len(intfspec) > 3:\n        name = intfspec[3]\n    else:\n        name = intfspec[0].split('.')[-1]\n    return name\n\n\ndef get_interface_groups(include_plugins=False):\n    \"\"\"Return a list of command groups.\n\n    Returns\n    -------\n    A list of tuples with the form (GROUP_NAME, GROUP_DESCRIPTION, COMMANDS).\n    \"\"\"\n    if include_plugins:\n        warnings.warn(\"Plugins are no longer supported.\", DeprecationWarning)\n\n    from .. import interface as _interfaces\n\n    grps = []\n    # auto detect all available interfaces and generate a function-based\n    # API from them\n    for _item in _interfaces.__dict__:\n        if not _item.startswith('_group_'):\n            continue\n        grp_name = _item[7:]\n        grp = getattr(_interfaces, _item)\n        grps.append((grp_name,) + grp)\n    return grps\n\n\ndef get_cmd_summaries(descriptions, groups, width=79):\n    \"\"\"Return summaries for the commands in `groups`.\n\n    Parameters\n    ----------\n    descriptions : dict\n        A map of group names to summaries.\n    groups : list of tuples\n        A list of groups and commands in the form described by\n        `get_interface_groups`.\n    width : int, optional\n        The maximum width of each line in the summary text.\n\n    Returns\n    -------\n    A list with a formatted entry for each command. The first command of each\n    group is preceded by an entry describing the group.\n    \"\"\"\n    cmd_summary = []\n    for grp in sorted(groups, key=lambda x: x[0]):\n        grp_descr = grp[1]\n        grp_cmds = descriptions[grp[0]]\n\n        cmd_summary.append('\\n*%s*\\n' % (grp_descr,))\n        for cd in grp_cmds:\n            cmd_summary.append('  %s\\n%s'\n                               % ((cd[0],\n                                   textwrap.fill(\n                                       cd[1].rstrip(' .'),\n                                       width - 5,\n                                       initial_indent=' ' * 6,\n                                       subsequent_indent=' ' * 6))))\n    return cmd_summary\n\n\ndef load_interface(spec):\n    \"\"\"Load and return the class for `spec`.\n\n    Parameters\n    ----------\n    spec : tuple\n        For a standard interface, the first item is the datalad source module\n        and the second object name for the interface.\n\n    Returns\n    -------\n    The interface class or, if importing the module fails, None.\n    \"\"\"\n    lgr.log(5, \"Importing module %s \", spec[0])\n    try:\n        mod = import_module(spec[0], package='datalad')\n    except Exception as e:\n        ce = CapturedException(e)\n        lgr.error(\"Internal error, cannot import interface '%s': %s\",\n                  spec[0], ce)\n        intf = None\n    else:\n        intf = getattr(mod, spec[1])\n    return intf\n\n\ndef get_cmd_doc(interface):\n    \"\"\"Return the documentation for the command defined by `interface`.\n\n    Parameters\n    ----------\n    interface : subclass of Interface\n    \"\"\"\n    intf_doc = '' if interface.__doc__ is None else interface.__doc__.strip()\n    if hasattr(interface, '_docs_'):\n        # expand docs\n        intf_doc = intf_doc.format(**interface._docs_)\n    return intf_doc\n\n\ndef dedent_docstring(text):\n    \"\"\"Remove uniform indentation from a multiline docstring\"\"\"\n    # Problem is that first line might often have no offset, so might\n    # need to be ignored from dedent call\n    if text is None:\n        return None\n    if not text.startswith(' '):\n        lines = text.split('\\n')\n        if len(lines) == 1:\n            # single line, no indentation, nothing to do\n            return text\n        text2 = '\\n'.join(lines[1:])\n        return lines[0] + \"\\n\" + textwrap.dedent(text2)\n    else:\n        return textwrap.dedent(text)\n\n\ndef alter_interface_docs_for_api(docs):\n    \"\"\"Apply modifications to interface docstrings for Python API use.\"\"\"\n    # central place to alter the impression of docstrings,\n    # like removing cmdline specific sections\n    if not docs:\n        return docs\n    docs = dedent_docstring(docs)\n    # clean cmdline sections\n    docs = re.sub(\n        r'\\|\\| CMDLINE \\>\\>.*?\\<\\< CMDLINE \\|\\|',\n        '',\n        docs,\n        flags=re.MULTILINE | re.DOTALL)\n    # clean cmdline in-line bits\n    docs = re.sub(\n        r'\\[CMD:\\s[^\\[\\]]*\\sCMD\\]',\n        '',\n        docs,\n        flags=re.MULTILINE | re.DOTALL)\n    docs = re.sub(\n        r'\\[PY:\\s([^\\[\\]]*)\\sPY\\]',\n        lambda match: match.group(1),\n        docs,\n        flags=re.MULTILINE)\n    # select only the python alternative from argument specifications\n    docs = re.sub(\n        r'``([a-zA-Z0-9_,.]+)\\|\\|([a-zA-Z0-9-,.]+)``',\n        lambda match: f'``{match.group(1)}``',\n        docs)\n    docs = re.sub(\n        r'\\|\\| PYTHON \\>\\>(.*?)\\<\\< PYTHON \\|\\|',\n        lambda match: match.group(1),\n        docs,\n        flags=re.MULTILINE | re.DOTALL)\n    if 'DATALAD_SPHINX_RUN' not in os.environ:\n        # remove :role:`...` RST markup for cmdline docs\n        docs = re.sub(\n            r':\\S+:`[^`]*`[\\\\]*',\n            lambda match: ':'.join(match.group(0).split(':')[2:]).strip('`\\\\'),\n            docs,\n            flags=re.MULTILINE | re.DOTALL)\n        # make the handbook doc references more accessible\n        # the URL is a redirect configured at readthedocs\n        docs = re.sub(\n            r'(handbook:[0-9]-[0-9]*)',\n            '\\\\1 (http://handbook.datalad.org/symbols)',\n            docs)\n    docs = re.sub(\n        r'^([ ]*)\\|\\| REFLOW \\>\\>\\n(.*?)\\<\\< REFLOW \\|\\|',\n        lambda match: textwrap.fill(match.group(2), subsequent_indent=match.group(1)),\n        docs,\n        flags=re.MULTILINE | re.DOTALL)\n    return docs\n\n\ndef is_api_arg(arg):\n    \"\"\"Return True if argument is our API argument or self or used for internal\n    purposes\n    \"\"\"\n    return arg != 'self' and not arg.startswith('_')\n\n\ndef update_docstring_with_parameters(func, params, prefix=None, suffix=None,\n                                     add_args=None):\n    \"\"\"Generate a useful docstring from a parameter spec\n\n    Amends any existing docstring of a callable with a textual\n    description of its parameters. The Parameter spec needs to match\n    the number and names of the callables arguments.\n    \"\"\"\n    from datalad.utils import getargspec\n\n    # get the signature\n    args, varargs, varkw, defaults = getargspec(func, include_kwonlyargs=True)\n    defaults = defaults or tuple()\n    if add_args:\n        add_argnames = sorted(add_args.keys())\n        args.extend(add_argnames)\n        defaults = defaults + tuple(add_args[k] for k in add_argnames)\n    ndefaults = len(defaults)\n    # start documentation with what the callable brings with it\n    doc = prefix if prefix else u''\n    if len(args) > 1:\n        if len(doc):\n            if not doc.endswith('\\n'):\n                doc += '\\n'\n            doc += '\\n'\n        doc += \"Parameters\\n----------\\n\"\n        for i, arg in enumerate(args):\n            if not is_api_arg(arg):\n                continue\n            # we need a parameter spec for each argument\n            if not arg in params:\n                raise ValueError(\"function has argument '%s' not described as a parameter\" % arg)\n            param = params[arg]\n            # validate the default -- to make sure that the parameter description is\n            # somewhat OK\n            defaults_idx = ndefaults - len(args) + i\n            if defaults_idx >= 0:\n                if param.constraints is not None:\n                    param.constraints(defaults[defaults_idx])\n            orig_docs = param._doc\n            param._doc = alter_interface_docs_for_api(param._doc)\n            doc += param.get_autodoc(\n                arg,\n                default=defaults[defaults_idx] if defaults_idx >= 0 else None,\n                has_default=defaults_idx >= 0)\n            param._doc = orig_docs\n            doc += '\\n'\n    doc += suffix if suffix else u\"\"\n    # assign the amended docs\n    func.__doc__ = doc\n    return func\n\n\n# TODO should export code_field and indicator, rather than have modes\n# TODO this should be a doc helper\ndef build_example(example, api='python'):\n    \"\"\"Build a code example.\n\n    Take a dict from a classes _example_ specification (list of dicts) and\n    build a string with an api or cmd example (for use in cmd help or\n    docstring).\n\n    Parameters\n    ----------\n    api : {'python', 'cmdline'}\n        If 'python', build Python example for docstring. If 'cmdline', build\n        cmd example.\n\n    Returns\n    -------\n    ex : str\n        Concatenated examples for the given class.\n    \"\"\"\n    if api == 'python' :\n        code_field='code_py'\n        indicator='>'\n    elif api == 'cmdline':\n        code_field='code_cmd'\n        indicator='%'\n    else:\n        raise ValueError(\"unknown API selection: {}\".format(api))\n    if code_field not in example:\n        # only show an example if it exist for the API\n        return ''\n    description = textwrap.fill(example.get('text'))\n    # this indent the code snippet to get it properly rendered as code\n    # we are not using textwrap.fill(), because it would not acknowledge\n    # any meaningful structure/formatting of code snippets. Instead, we\n    # maintain line content as is.\n    code = dedent_docstring(example.get(code_field))\n    needs_indicator = not code.startswith(indicator)\n    code = textwrap.indent(code, ' ' * (5 if needs_indicator else 3)).lstrip()\n\n    ex = \"\"\"{}::\\n\\n   {}{}\\n\\n\"\"\".format(\n        description,\n        # disable automatic prefixing, if the example already has one\n        # this enables providing more complex examples without having\n        # to infer its inner structure\n        '{} '.format(indicator)\n        if needs_indicator\n        # maintain spacing to avoid undesired relative indentation\n        else '',\n        code)\n\n    return ex\n\n\ndef update_docstring_with_examples(cls_doc, ex):\n    \"\"\"Update a commands docstring with examples.\n\n    Take _examples_ of a command, build the Python examples, and append\n    them to the docstring.\n\n    Parameters\n    ----------\n    cls_doc: str\n      docstring\n    ex: list\n      list of dicts with examples\n    \"\"\"\n    from textwrap import indent\n    if len(cls_doc):\n        cls_doc += \"\\n\"\n    cls_doc += \"    Examples\\n    --------\\n\"\n    # loop though provided examples\n    for example in ex:\n        cls_doc += indent(build_example(example, api='python'), ' '*4)\n\n    return cls_doc\n\n\ndef build_doc(cls, **kwargs):\n    \"\"\"Decorator to build docstrings for datalad commands\n\n    It's intended to decorate the class, the __call__-method of which is the\n    actual command. It expects that __call__-method to be decorated by\n    eval_results.\n\n    Note that values for any `eval_params` keys in `cls._params_` are\n    ignored.  This means one class may extend another's `_params_`\n    without worrying about filtering out `eval_params`.\n\n    Parameters\n    ----------\n    cls: Interface\n      DataLad command implementation\n    \"\"\"\n    if datalad.in_librarymode():\n        lgr.debug(\"Not assembling DataLad API docs in libary-mode\")\n        return cls\n\n    # Note, that this is a class decorator, which is executed only once when the\n    # class is imported. It builds the docstring for the class' __call__ method\n    # and returns the original class.\n    #\n    # This is because a decorator for the actual function would not be able to\n    # behave like this. To build the docstring we need to access the attribute\n    # _params of the class. From within a function decorator we cannot do this\n    # during import time, since the class is being built in this very moment and\n    # is not yet available in the module. And if we do it from within the part\n    # of a function decorator, that is executed when the function is called, we\n    # would need to actually call the command once in order to build this\n    # docstring.\n\n    lgr.debug(\"Building doc for %s\", cls)\n\n    cls_doc = cls.__doc__\n    if hasattr(cls, '_docs_'):\n        # expand docs\n        cls_doc = cls_doc.format(**cls._docs_)\n    # get examples\n    ex = getattr(cls, '_examples_', [])\n    if ex:\n        cls_doc = update_docstring_with_examples(cls_doc, ex)\n\n    call_doc = None\n    # suffix for update_docstring_with_parameters:\n    if cls.__call__.__doc__:\n        call_doc = cls.__call__.__doc__\n\n    # build standard doc and insert eval_doc\n    spec = getattr(cls, '_params_', dict())\n\n\n    # update class attributes that may override defaults\n    if not _has_eval_results_call(cls):\n        add_args = None\n    else:\n        # defaults for all common parameters are guaranteed to be available\n        # from the class\n        add_args = {k: getattr(cls, k) for k in eval_params}\n\n    # ATTN: An important consequence of this update() call is that it\n    # fulfills the docstring's promise of overriding any existing\n    # values for eval_params keys in _params_.\n    #\n\n    # get docs for eval_results parameters:\n    spec.update(eval_params)\n\n    update_docstring_with_parameters(\n        cls.__call__, spec,\n        prefix=alter_interface_docs_for_api(cls_doc),\n        suffix=alter_interface_docs_for_api(call_doc),\n        add_args=add_args\n    )\n\n    if hasattr(cls.__call__, '_dataset_method'):\n        cls.__call__._dataset_method.__doc__ = cls.__call__.__doc__\n\n    # return original\n    return cls\n\n\nclass Interface(ABC):\n    '''Abstract base class for DataLad command implementations\n\n    Any DataLad command implementation must be derived from this class. The\n    code snippet below shows a complete sketch of a Python class with such an\n    implementation.\n\n    Importantly, no instances of command classes will created. Instead the main\n    entry point is a static ``__call__()`` method, which must be implemented\n    for any command. It is incorporated as a function in :mod:`datalad.api`, by\n    default under the name of the file the implementation resides (e.g.,\n    ``command`` for a ``command.py`` file).  Therefore the file should have a\n    name that is a syntax-compliant function name. The default naming rule can\n    be overwritten with an explicit alternative name (see\n    :func:`datalad.interface.base.get_api_name`).\n\n    For commands implementing functionality that is operating on DataLad\n    datasets, a command can be also be bound to the\n    :class:`~datalad.distribution.dataset.Dataset` class as a method using\n    the ``@datasetmethod`` decorator, under the specified name.\n\n    Any ``__call__()`` implementation should be decorated with\n    :func:`datalad.interface.utils.eval_results`. This adds support for\n    standard result processing, and a range of common command parameters that\n    do not need to be manually added to the signature of ``__call__()``. Any\n    implementation decorated in this way should be implemented as a generator,\n    and ``yield`` :ref:`result records <chap_design_result_records>`.\n\n    Any argument or keyword argument that appears in the signature of\n    ``__call__()`` must have a matching item in :attr:`Interface._params_`.\n    The dictionary maps argument names to\n    :class:`datalad.support.param.Parameter` specifications. The specification\n    contain CLI argument declarations, value constraint and data type\n    conversation specifications, documentation, and optional\n    ``argparse``-specific arguments for CLI parser construction.\n\n    The class decorator :func:`datalad.interface.base.build_doc` inspects an\n    :class:`Interface` implementation, and builds a standard docstring from\n    various sources of structured information within the class (also see\n    below). The documentation is automatically tuned differently, depending on\n    the target API (Python vs CLI).\n\n    .. code:: python\n\n        @build_doc\n        class ExampleCommand(Interface):\n            \"\"\"SHORT DESCRIPTION\n\n            LONG DESCRIPTION\n            ...\n            \"\"\"\n\n            # COMMAND PARAMETER DEFINITIONS\n            _params_ = dict(\n                example=Parameter(\n                    args=(\"--example\",),\n                    doc=\"\"\"Parameter description....\"\"\",\n                    constraints=...),\n                ...\n                )\n            )\n\n            # RESULT PARAMETER OVERRIDES\n            return_type= 'list'\n            ...\n\n            # USAGE EXAMPLES\n            _examples_ = [\n                dict(text=\"Example description...\",\n                     code_py=\"Example Python code...\",\n                     code_cmd=\"Example shell code ...\"),\n                ...\n            ]\n\n            @staticmethod\n            @datasetmethod(name='example_command')\n            @eval_results\n            def __call__(example=None, ...):\n                ...\n\n                yield dict(...)\n\n    The basic implementation setup described above can be customized for\n    individual commands in various way that alter the behavior and\n    presentation of a specific command. The following overview uses\n    the code comment markers in the above snippet to illustrate where\n    in the class implementation these adjustments can be made.\n\n    (SHORT/LONG) DESCRIPTION\n\n    ``Interface.short_description`` can be defined to provide an\n    explicit short description to be used in documentation and help output,\n    replacing the auto-generated extract from the first line of the full\n    description.\n\n    COMMAND PARAMETER DEFINITIONS\n\n    When a parameter specification declares ``Parameter(args=tuple(), ...)``,\n    i.e. no arguments specified, it will be ignored by the CLI. Likewise, any\n    ``Parameter`` specification for which :func:`is_api_arg` returns ``False``\n    will also be ignored by the CLI. Additionally, any such parameter will\n    not be added to the parameter description list in the Python docstring.\n\n    RESULT PARAMETER OVERRIDES\n\n    The :func:`datalad.interface.utils.eval_results` decorator automatically\n    add a range of additional arguments to a command, which are defined in\n    :py:data:`datalad.interface.common_opts.eval_params`. For any such\n    parameter an Interface implementation can define an interface-specific\n    default value, by declaring a class member with the respective parameter\n    name and the desired default as its assigned value. This feature can be\n    used to tune the default command behavior, for example, with respect to the\n    default result rendering style, or its error behavior.\n\n    In addition to the common parameters of the Python API, an additional\n    ``Interface.result_renderer_cmdline`` can be defined, in order to\n    instruct the CLI to prefer the specified alternative result renderer\n    over an ``Interface.result_renderer`` specification.\n\n    USAGE EXAMPLES\n\n    Any number of usage examples can be described in an ``_examples_`` list\n    class attribute. Such an example contains a description, and code examples\n    for Python and CLI.\n    '''\n    _params_ = {}\n\n    @abstractmethod\n    def __call__():\n        \"\"\"Must be implemented by any command\"\"\"\n\n    # https://github.com/datalad/datalad/issues/6376\n    @classmethod\n    def get_refds_path(cls, dataset):\n        \"\"\"Return a resolved reference dataset path from a `dataset` argument\n\n        .. deprecated:: 0.16\n           Use ``require_dataset()`` instead.\n        \"\"\"\n        # theoretically a dataset could come in as a relative path -> resolve\n        if dataset is None:\n            return dataset\n        refds_path = dataset.path if isinstance(dataset, Dataset) \\\n            else Dataset(dataset).path\n        if refds_path:\n            refds_path = str(resolve_path(refds_path))\n        return refds_path\n\n\n# pull all defaults from all eval_results() related parameters and assign them\n# as attributes to the class, which then becomes the one place to query for\n# default and potential overrides\nfor k, p in eval_params.items():\n    setattr(Interface,\n            # name is always given\n            k,\n            # but there may be no default (rather unlikely, though)\n            p.cmd_kwargs.get('default', None))\n\n\ndef get_allargs_as_kwargs(call, args, kwargs):\n    \"\"\"Generate a kwargs dict from a call signature and ``*args``, ``**kwargs``\n\n    Basically resolving the argnames for all positional arguments, and\n    resolving the defaults for all kwargs that are not given in a kwargs\n    dict\n    \"\"\"\n    from datalad.utils import getargspec\n    argspec = getargspec(call, include_kwonlyargs=True)\n    defaults = argspec.defaults\n    nargs = len(argspec.args)\n    defaults = defaults or []  # ensure it is a list and not None\n    assert (nargs >= len(defaults))\n    # map any args to their name\n    argmap = list(zip(argspec.args[:len(args)], args))\n    kwargs_ = OrderedDict(argmap)\n    # map defaults of kwargs to their names (update below)\n    for k, v in zip(argspec.args[-len(defaults):], defaults):\n        if k not in kwargs_:\n            kwargs_[k] = v\n    # update with provided kwarg args\n    kwargs_.update(kwargs)\n    # XXX we cannot assert the following, because our own highlevel\n    # API commands support more kwargs than what is discoverable\n    # from their signature...\n    #assert (nargs == len(kwargs_))\n    return kwargs_\n\n\n# Only needed to support command implementations before the introduction\n# of @eval_results\ndef _has_eval_results_call(cls):\n    \"\"\"Return True if cls has a __call__ decorated with @eval_results\n    \"\"\"\n    return getattr(getattr(cls, '__call__', None), '_eval_results', False)\n",
            "new_file_content": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 et:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"High-level interface generation\n\n\"\"\"\n\n__docformat__ = 'restructuredtext'\n\nimport logging\n\nlgr = logging.getLogger('datalad.interface.base')\n\nimport os\nimport re\nimport textwrap\nimport warnings\nfrom abc import (\n    ABC,\n    abstractmethod,\n)\nfrom collections import OrderedDict\nfrom importlib import import_module\n\nimport datalad\nfrom datalad.distribution.dataset import (\n    Dataset,\n    resolve_path,\n)\nfrom datalad.interface.common_opts import eval_params\nfrom datalad.support.exceptions import CapturedException\n\ndefault_logchannels = {\n    '': 'debug',\n    'ok': 'debug',\n    'notneeded': 'debug',\n    'impossible': 'warning',\n    'error': 'error',\n}\n\n\ndef get_api_name(intfspec):\n    \"\"\"Given an interface specification return an API name for it\"\"\"\n    if len(intfspec) > 3:\n        name = intfspec[3]\n    else:\n        name = intfspec[0].split('.')[-1]\n    return name\n\n\ndef get_interface_groups(include_plugins=False):\n    \"\"\"Return a list of command groups.\n\n    Returns\n    -------\n    A list of tuples with the form (GROUP_NAME, GROUP_DESCRIPTION, COMMANDS).\n    \"\"\"\n    if include_plugins:\n        warnings.warn(\"Plugins are no longer supported.\", DeprecationWarning)\n\n    from .. import interface as _interfaces\n\n    grps = []\n    # auto detect all available interfaces and generate a function-based\n    # API from them\n    for _item in _interfaces.__dict__:\n        if not _item.startswith('_group_'):\n            continue\n        grp_name = _item[7:]\n        grp = getattr(_interfaces, _item)\n        grps.append((grp_name,) + grp)\n    return grps\n\n\ndef get_cmd_summaries(descriptions, groups, width=79):\n    \"\"\"Return summaries for the commands in `groups`.\n\n    Parameters\n    ----------\n    descriptions : dict\n        A map of group names to summaries.\n    groups : list of tuples\n        A list of groups and commands in the form described by\n        `get_interface_groups`.\n    width : int, optional\n        The maximum width of each line in the summary text.\n\n    Returns\n    -------\n    A list with a formatted entry for each command. The first command of each\n    group is preceded by an entry describing the group.\n    \"\"\"\n    cmd_summary = []\n    for grp in sorted(groups, key=lambda x: x[0]):\n        grp_descr = grp[1]\n        grp_cmds = descriptions[grp[0]]\n\n        cmd_summary.append('\\n*%s*\\n' % (grp_descr,))\n        for cd in grp_cmds:\n            cmd_summary.append('  %s\\n%s'\n                               % ((cd[0],\n                                   textwrap.fill(\n                                       cd[1].rstrip(' .'),\n                                       width - 5,\n                                       initial_indent=' ' * 6,\n                                       subsequent_indent=' ' * 6))))\n    return cmd_summary\n\n\ndef load_interface(spec):\n    \"\"\"Load and return the class for `spec`.\n\n    Parameters\n    ----------\n    spec : tuple\n        For a standard interface, the first item is the datalad source module\n        and the second object name for the interface.\n\n    Returns\n    -------\n    The interface class or, if importing the module fails, None.\n    \"\"\"\n    lgr.log(5, \"Importing module %s \", spec[0])\n    try:\n        mod = import_module(spec[0], package='datalad')\n    except Exception as e:\n        ce = CapturedException(e)\n        lgr.error(\"Internal error, cannot import interface '%s': %s\",\n                  spec[0], ce)\n        intf = None\n    else:\n        intf = getattr(mod, spec[1])\n    return intf\n\n\ndef get_cmd_doc(interface):\n    \"\"\"Return the documentation for the command defined by `interface`.\n\n    Parameters\n    ----------\n    interface : subclass of Interface\n    \"\"\"\n    intf_doc = '' if interface.__doc__ is None else interface.__doc__.strip()\n    if hasattr(interface, '_docs_'):\n        # expand docs\n        intf_doc = intf_doc.format(**interface._docs_)\n    return intf_doc\n\n\ndef dedent_docstring(text):\n    \"\"\"Remove uniform indentation from a multiline docstring\"\"\"\n    # Problem is that first line might often have no offset, so might\n    # need to be ignored from dedent call\n    if text is None:\n        return None\n    if not text.startswith(' '):\n        lines = text.split('\\n')\n        if len(lines) == 1:\n            # single line, no indentation, nothing to do\n            return text\n        text2 = '\\n'.join(lines[1:])\n        return lines[0] + \"\\n\" + textwrap.dedent(text2)\n    else:\n        return textwrap.dedent(text)\n\n\ndef alter_interface_docs_for_api(docs):\n    \"\"\"Apply modifications to interface docstrings for Python API use.\"\"\"\n    # central place to alter the impression of docstrings,\n    # like removing cmdline specific sections\n    if not docs:\n        return docs\n    docs = dedent_docstring(docs)\n    # clean cmdline sections\n    docs = re.sub(\n        r'\\|\\| CMDLINE \\>\\>.*?\\<\\< CMDLINE \\|\\|',\n        '',\n        docs,\n        flags=re.MULTILINE | re.DOTALL)\n    # clean cmdline in-line bits\n    docs = re.sub(\n        r'\\[CMD:\\s.*?\\sCMD\\]',\n        '',\n        docs,\n        flags=re.MULTILINE | re.DOTALL)\n    docs = re.sub(\n        r'\\[PY:\\s(.*?)\\sPY\\]',\n        lambda match: match.group(1),\n        docs,\n        flags=re.MULTILINE | re.DOTALL)\n    # select only the python alternative from argument specifications\n    docs = re.sub(\n        r'``([a-zA-Z0-9_,.]+)\\|\\|([a-zA-Z0-9-,.]+)``',\n        lambda match: f'``{match.group(1)}``',\n        docs)\n    docs = re.sub(\n        r'\\|\\| PYTHON \\>\\>(.*?)\\<\\< PYTHON \\|\\|',\n        lambda match: match.group(1),\n        docs,\n        flags=re.MULTILINE | re.DOTALL)\n    if 'DATALAD_SPHINX_RUN' not in os.environ:\n        # remove :role:`...` RST markup for cmdline docs\n        docs = re.sub(\n            r':\\S+:`[^`]*`[\\\\]*',\n            lambda match: ':'.join(match.group(0).split(':')[2:]).strip('`\\\\'),\n            docs,\n            flags=re.MULTILINE | re.DOTALL)\n        # make the handbook doc references more accessible\n        # the URL is a redirect configured at readthedocs\n        docs = re.sub(\n            r'(handbook:[0-9]-[0-9]*)',\n            '\\\\1 (http://handbook.datalad.org/symbols)',\n            docs)\n    docs = re.sub(\n        r'^([ ]*)\\|\\| REFLOW \\>\\>\\n(.*?)\\<\\< REFLOW \\|\\|',\n        lambda match: textwrap.fill(match.group(2), subsequent_indent=match.group(1)),\n        docs,\n        flags=re.MULTILINE | re.DOTALL)\n    return docs\n\n\ndef is_api_arg(arg):\n    \"\"\"Return True if argument is our API argument or self or used for internal\n    purposes\n    \"\"\"\n    return arg != 'self' and not arg.startswith('_')\n\n\ndef update_docstring_with_parameters(func, params, prefix=None, suffix=None,\n                                     add_args=None):\n    \"\"\"Generate a useful docstring from a parameter spec\n\n    Amends any existing docstring of a callable with a textual\n    description of its parameters. The Parameter spec needs to match\n    the number and names of the callables arguments.\n    \"\"\"\n    from datalad.utils import getargspec\n\n    # get the signature\n    args, varargs, varkw, defaults = getargspec(func, include_kwonlyargs=True)\n    defaults = defaults or tuple()\n    if add_args:\n        add_argnames = sorted(add_args.keys())\n        args.extend(add_argnames)\n        defaults = defaults + tuple(add_args[k] for k in add_argnames)\n    ndefaults = len(defaults)\n    # start documentation with what the callable brings with it\n    doc = prefix if prefix else u''\n    if len(args) > 1:\n        if len(doc):\n            if not doc.endswith('\\n'):\n                doc += '\\n'\n            doc += '\\n'\n        doc += \"Parameters\\n----------\\n\"\n        for i, arg in enumerate(args):\n            if not is_api_arg(arg):\n                continue\n            # we need a parameter spec for each argument\n            if not arg in params:\n                raise ValueError(\"function has argument '%s' not described as a parameter\" % arg)\n            param = params[arg]\n            # validate the default -- to make sure that the parameter description is\n            # somewhat OK\n            defaults_idx = ndefaults - len(args) + i\n            if defaults_idx >= 0:\n                if param.constraints is not None:\n                    param.constraints(defaults[defaults_idx])\n            orig_docs = param._doc\n            param._doc = alter_interface_docs_for_api(param._doc)\n            doc += param.get_autodoc(\n                arg,\n                default=defaults[defaults_idx] if defaults_idx >= 0 else None,\n                has_default=defaults_idx >= 0)\n            param._doc = orig_docs\n            doc += '\\n'\n    doc += suffix if suffix else u\"\"\n    # assign the amended docs\n    func.__doc__ = doc\n    return func\n\n\n# TODO should export code_field and indicator, rather than have modes\n# TODO this should be a doc helper\ndef build_example(example, api='python'):\n    \"\"\"Build a code example.\n\n    Take a dict from a classes _example_ specification (list of dicts) and\n    build a string with an api or cmd example (for use in cmd help or\n    docstring).\n\n    Parameters\n    ----------\n    api : {'python', 'cmdline'}\n        If 'python', build Python example for docstring. If 'cmdline', build\n        cmd example.\n\n    Returns\n    -------\n    ex : str\n        Concatenated examples for the given class.\n    \"\"\"\n    if api == 'python' :\n        code_field='code_py'\n        indicator='>'\n    elif api == 'cmdline':\n        code_field='code_cmd'\n        indicator='%'\n    else:\n        raise ValueError(\"unknown API selection: {}\".format(api))\n    if code_field not in example:\n        # only show an example if it exist for the API\n        return ''\n    description = textwrap.fill(example.get('text'))\n    # this indent the code snippet to get it properly rendered as code\n    # we are not using textwrap.fill(), because it would not acknowledge\n    # any meaningful structure/formatting of code snippets. Instead, we\n    # maintain line content as is.\n    code = dedent_docstring(example.get(code_field))\n    needs_indicator = not code.startswith(indicator)\n    code = textwrap.indent(code, ' ' * (5 if needs_indicator else 3)).lstrip()\n\n    ex = \"\"\"{}::\\n\\n   {}{}\\n\\n\"\"\".format(\n        description,\n        # disable automatic prefixing, if the example already has one\n        # this enables providing more complex examples without having\n        # to infer its inner structure\n        '{} '.format(indicator)\n        if needs_indicator\n        # maintain spacing to avoid undesired relative indentation\n        else '',\n        code)\n\n    return ex\n\n\ndef update_docstring_with_examples(cls_doc, ex):\n    \"\"\"Update a commands docstring with examples.\n\n    Take _examples_ of a command, build the Python examples, and append\n    them to the docstring.\n\n    Parameters\n    ----------\n    cls_doc: str\n      docstring\n    ex: list\n      list of dicts with examples\n    \"\"\"\n    from textwrap import indent\n    if len(cls_doc):\n        cls_doc += \"\\n\"\n    cls_doc += \"    Examples\\n    --------\\n\"\n    # loop though provided examples\n    for example in ex:\n        cls_doc += indent(build_example(example, api='python'), ' '*4)\n\n    return cls_doc\n\n\ndef build_doc(cls, **kwargs):\n    \"\"\"Decorator to build docstrings for datalad commands\n\n    It's intended to decorate the class, the __call__-method of which is the\n    actual command. It expects that __call__-method to be decorated by\n    eval_results.\n\n    Note that values for any `eval_params` keys in `cls._params_` are\n    ignored.  This means one class may extend another's `_params_`\n    without worrying about filtering out `eval_params`.\n\n    Parameters\n    ----------\n    cls: Interface\n      DataLad command implementation\n    \"\"\"\n    if datalad.in_librarymode():\n        lgr.debug(\"Not assembling DataLad API docs in libary-mode\")\n        return cls\n\n    # Note, that this is a class decorator, which is executed only once when the\n    # class is imported. It builds the docstring for the class' __call__ method\n    # and returns the original class.\n    #\n    # This is because a decorator for the actual function would not be able to\n    # behave like this. To build the docstring we need to access the attribute\n    # _params of the class. From within a function decorator we cannot do this\n    # during import time, since the class is being built in this very moment and\n    # is not yet available in the module. And if we do it from within the part\n    # of a function decorator, that is executed when the function is called, we\n    # would need to actually call the command once in order to build this\n    # docstring.\n\n    lgr.debug(\"Building doc for %s\", cls)\n\n    cls_doc = cls.__doc__\n    if hasattr(cls, '_docs_'):\n        # expand docs\n        cls_doc = cls_doc.format(**cls._docs_)\n    # get examples\n    ex = getattr(cls, '_examples_', [])\n    if ex:\n        cls_doc = update_docstring_with_examples(cls_doc, ex)\n\n    call_doc = None\n    # suffix for update_docstring_with_parameters:\n    if cls.__call__.__doc__:\n        call_doc = cls.__call__.__doc__\n\n    # build standard doc and insert eval_doc\n    spec = getattr(cls, '_params_', dict())\n\n\n    # update class attributes that may override defaults\n    if not _has_eval_results_call(cls):\n        add_args = None\n    else:\n        # defaults for all common parameters are guaranteed to be available\n        # from the class\n        add_args = {k: getattr(cls, k) for k in eval_params}\n\n    # ATTN: An important consequence of this update() call is that it\n    # fulfills the docstring's promise of overriding any existing\n    # values for eval_params keys in _params_.\n    #\n\n    # get docs for eval_results parameters:\n    spec.update(eval_params)\n\n    update_docstring_with_parameters(\n        cls.__call__, spec,\n        prefix=alter_interface_docs_for_api(cls_doc),\n        suffix=alter_interface_docs_for_api(call_doc),\n        add_args=add_args\n    )\n\n    if hasattr(cls.__call__, '_dataset_method'):\n        cls.__call__._dataset_method.__doc__ = cls.__call__.__doc__\n\n    # return original\n    return cls\n\n\nclass Interface(ABC):\n    '''Abstract base class for DataLad command implementations\n\n    Any DataLad command implementation must be derived from this class. The\n    code snippet below shows a complete sketch of a Python class with such an\n    implementation.\n\n    Importantly, no instances of command classes will created. Instead the main\n    entry point is a static ``__call__()`` method, which must be implemented\n    for any command. It is incorporated as a function in :mod:`datalad.api`, by\n    default under the name of the file the implementation resides (e.g.,\n    ``command`` for a ``command.py`` file).  Therefore the file should have a\n    name that is a syntax-compliant function name. The default naming rule can\n    be overwritten with an explicit alternative name (see\n    :func:`datalad.interface.base.get_api_name`).\n\n    For commands implementing functionality that is operating on DataLad\n    datasets, a command can be also be bound to the\n    :class:`~datalad.distribution.dataset.Dataset` class as a method using\n    the ``@datasetmethod`` decorator, under the specified name.\n\n    Any ``__call__()`` implementation should be decorated with\n    :func:`datalad.interface.utils.eval_results`. This adds support for\n    standard result processing, and a range of common command parameters that\n    do not need to be manually added to the signature of ``__call__()``. Any\n    implementation decorated in this way should be implemented as a generator,\n    and ``yield`` :ref:`result records <chap_design_result_records>`.\n\n    Any argument or keyword argument that appears in the signature of\n    ``__call__()`` must have a matching item in :attr:`Interface._params_`.\n    The dictionary maps argument names to\n    :class:`datalad.support.param.Parameter` specifications. The specification\n    contain CLI argument declarations, value constraint and data type\n    conversation specifications, documentation, and optional\n    ``argparse``-specific arguments for CLI parser construction.\n\n    The class decorator :func:`datalad.interface.base.build_doc` inspects an\n    :class:`Interface` implementation, and builds a standard docstring from\n    various sources of structured information within the class (also see\n    below). The documentation is automatically tuned differently, depending on\n    the target API (Python vs CLI).\n\n    .. code:: python\n\n        @build_doc\n        class ExampleCommand(Interface):\n            \"\"\"SHORT DESCRIPTION\n\n            LONG DESCRIPTION\n            ...\n            \"\"\"\n\n            # COMMAND PARAMETER DEFINITIONS\n            _params_ = dict(\n                example=Parameter(\n                    args=(\"--example\",),\n                    doc=\"\"\"Parameter description....\"\"\",\n                    constraints=...),\n                ...\n                )\n            )\n\n            # RESULT PARAMETER OVERRIDES\n            return_type= 'list'\n            ...\n\n            # USAGE EXAMPLES\n            _examples_ = [\n                dict(text=\"Example description...\",\n                     code_py=\"Example Python code...\",\n                     code_cmd=\"Example shell code ...\"),\n                ...\n            ]\n\n            @staticmethod\n            @datasetmethod(name='example_command')\n            @eval_results\n            def __call__(example=None, ...):\n                ...\n\n                yield dict(...)\n\n    The basic implementation setup described above can be customized for\n    individual commands in various way that alter the behavior and\n    presentation of a specific command. The following overview uses\n    the code comment markers in the above snippet to illustrate where\n    in the class implementation these adjustments can be made.\n\n    (SHORT/LONG) DESCRIPTION\n\n    ``Interface.short_description`` can be defined to provide an\n    explicit short description to be used in documentation and help output,\n    replacing the auto-generated extract from the first line of the full\n    description.\n\n    COMMAND PARAMETER DEFINITIONS\n\n    When a parameter specification declares ``Parameter(args=tuple(), ...)``,\n    i.e. no arguments specified, it will be ignored by the CLI. Likewise, any\n    ``Parameter`` specification for which :func:`is_api_arg` returns ``False``\n    will also be ignored by the CLI. Additionally, any such parameter will\n    not be added to the parameter description list in the Python docstring.\n\n    RESULT PARAMETER OVERRIDES\n\n    The :func:`datalad.interface.utils.eval_results` decorator automatically\n    add a range of additional arguments to a command, which are defined in\n    :py:data:`datalad.interface.common_opts.eval_params`. For any such\n    parameter an Interface implementation can define an interface-specific\n    default value, by declaring a class member with the respective parameter\n    name and the desired default as its assigned value. This feature can be\n    used to tune the default command behavior, for example, with respect to the\n    default result rendering style, or its error behavior.\n\n    In addition to the common parameters of the Python API, an additional\n    ``Interface.result_renderer_cmdline`` can be defined, in order to\n    instruct the CLI to prefer the specified alternative result renderer\n    over an ``Interface.result_renderer`` specification.\n\n    USAGE EXAMPLES\n\n    Any number of usage examples can be described in an ``_examples_`` list\n    class attribute. Such an example contains a description, and code examples\n    for Python and CLI.\n    '''\n    _params_ = {}\n\n    @abstractmethod\n    def __call__():\n        \"\"\"Must be implemented by any command\"\"\"\n\n    # https://github.com/datalad/datalad/issues/6376\n    @classmethod\n    def get_refds_path(cls, dataset):\n        \"\"\"Return a resolved reference dataset path from a `dataset` argument\n\n        .. deprecated:: 0.16\n           Use ``require_dataset()`` instead.\n        \"\"\"\n        # theoretically a dataset could come in as a relative path -> resolve\n        if dataset is None:\n            return dataset\n        refds_path = dataset.path if isinstance(dataset, Dataset) \\\n            else Dataset(dataset).path\n        if refds_path:\n            refds_path = str(resolve_path(refds_path))\n        return refds_path\n\n\n# pull all defaults from all eval_results() related parameters and assign them\n# as attributes to the class, which then becomes the one place to query for\n# default and potential overrides\nfor k, p in eval_params.items():\n    setattr(Interface,\n            # name is always given\n            k,\n            # but there may be no default (rather unlikely, though)\n            p.cmd_kwargs.get('default', None))\n\n\ndef get_allargs_as_kwargs(call, args, kwargs):\n    \"\"\"Generate a kwargs dict from a call signature and ``*args``, ``**kwargs``\n\n    Basically resolving the argnames for all positional arguments, and\n    resolving the defaults for all kwargs that are not given in a kwargs\n    dict\n    \"\"\"\n    from datalad.utils import getargspec\n    argspec = getargspec(call, include_kwonlyargs=True)\n    defaults = argspec.defaults\n    nargs = len(argspec.args)\n    defaults = defaults or []  # ensure it is a list and not None\n    assert (nargs >= len(defaults))\n    # map any args to their name\n    argmap = list(zip(argspec.args[:len(args)], args))\n    kwargs_ = OrderedDict(argmap)\n    # map defaults of kwargs to their names (update below)\n    for k, v in zip(argspec.args[-len(defaults):], defaults):\n        if k not in kwargs_:\n            kwargs_[k] = v\n    # update with provided kwarg args\n    kwargs_.update(kwargs)\n    # XXX we cannot assert the following, because our own highlevel\n    # API commands support more kwargs than what is discoverable\n    # from their signature...\n    #assert (nargs == len(kwargs_))\n    return kwargs_\n\n\n# Only needed to support command implementations before the introduction\n# of @eval_results\ndef _has_eval_results_call(cls):\n    \"\"\"Return True if cls has a __call__ decorated with @eval_results\n    \"\"\"\n    return getattr(getattr(cls, '__call__', None), '_eval_results', False)\n",
            "header": {
                "file": {
                    "path": "datalad/interface/base.py"
                },
                "misc_line": null
            },
            "index_line": {
                "old_commit_hash": "6dc512586",
                "new_commit_hash": "4bdbdd17e",
                "mode": "100644"
            },
            "is_binary_file": false,
            "binary_line": null,
            "minus_file": {
                "path": "a/datalad/interface/base.py"
            },
            "plus_file": {
                "path": "b/datalad/interface/base.py"
            },
            "hunks": [
                {
                    "descriptor": {
                        "old_range": {
                            "start": 184,
                            "length": 15
                        },
                        "new_range": {
                            "start": 184,
                            "length": 15
                        },
                        "section": "def alter_interface_docs_for_api(docs):"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "        flags=re.MULTILINE | re.DOTALL)",
                                "type": "context"
                            },
                            {
                                "content": "    # clean cmdline in-line bits",
                                "type": "context"
                            },
                            {
                                "content": "    docs = re.sub(",
                                "type": "context"
                            },
                            {
                                "content": "        r'\\[CMD:\\s[^\\[\\]]*\\sCMD\\]',",
                                "type": "deleted"
                            },
                            {
                                "content": "        r'\\[CMD:\\s.*?\\sCMD\\]',",
                                "type": "added"
                            },
                            {
                                "content": "        '',",
                                "type": "context"
                            },
                            {
                                "content": "        docs,",
                                "type": "context"
                            },
                            {
                                "content": "        flags=re.MULTILINE | re.DOTALL)",
                                "type": "context"
                            },
                            {
                                "content": "    docs = re.sub(",
                                "type": "context"
                            },
                            {
                                "content": "        r'\\[PY:\\s([^\\[\\]]*)\\sPY\\]',",
                                "type": "deleted"
                            },
                            {
                                "content": "        r'\\[PY:\\s(.*?)\\sPY\\]',",
                                "type": "added"
                            },
                            {
                                "content": "        lambda match: match.group(1),",
                                "type": "context"
                            },
                            {
                                "content": "        docs,",
                                "type": "context"
                            },
                            {
                                "content": "        flags=re.MULTILINE)",
                                "type": "deleted"
                            },
                            {
                                "content": "        flags=re.MULTILINE | re.DOTALL)",
                                "type": "added"
                            },
                            {
                                "content": "    # select only the python alternative from argument specifications",
                                "type": "context"
                            },
                            {
                                "content": "    docs = re.sub(",
                                "type": "context"
                            },
                            {
                                "content": "        r'``([a-zA-Z0-9_,.]+)\\|\\|([a-zA-Z0-9-,.]+)``',",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [
                        {
                            "file_name": "datalad/interface/base.py",
                            "type": "function",
                            "name": "alter_interface_docs_for_api",
                            "content": "def alter_interface_docs_for_api(docs):\n    \"\"\"Apply modifications to interface docstrings for Python API use.\"\"\"\n    # central place to alter the impression of docstrings,\n    # like removing cmdline specific sections\n    if not docs:\n        return docs\n    docs = dedent_docstring(docs)\n    # clean cmdline sections\n    docs = re.sub(\n        r'\\|\\| CMDLINE \\>\\>.*?\\<\\< CMDLINE \\|\\|',\n        '',\n        docs,\n        flags=re.MULTILINE | re.DOTALL)\n    # clean cmdline in-line bits\n    docs = re.sub(\n        r'\\[CMD:\\s[^\\[\\]]*\\sCMD\\]',\n        '',\n        docs,\n        flags=re.MULTILINE | re.DOTALL)\n    docs = re.sub(\n        r'\\[PY:\\s([^\\[\\]]*)\\sPY\\]',\n        lambda match: match.group(1),\n        docs,\n        flags=re.MULTILINE)\n    # select only the python alternative from argument specifications\n    docs = re.sub(\n        r'``([a-zA-Z0-9_,.]+)\\|\\|([a-zA-Z0-9-,.]+)``',\n        lambda match: f'``{match.group(1)}``',\n        docs)\n    docs = re.sub(\n        r'\\|\\| PYTHON \\>\\>(.*?)\\<\\< PYTHON \\|\\|',\n        lambda match: match.group(1),\n        docs,\n        flags=re.MULTILINE | re.DOTALL)\n    if 'DATALAD_SPHINX_RUN' not in os.environ:\n        # remove :role:`...` RST markup for cmdline docs\n        docs = re.sub(\n            r':\\S+:`[^`]*`[\\\\]*',\n            lambda match: ':'.join(match.group(0).split(':')[2:]).strip('`\\\\'),\n            docs,\n            flags=re.MULTILINE | re.DOTALL)\n        # make the handbook doc references more accessible\n        # the URL is a redirect configured at readthedocs\n        docs = re.sub(\n            r'(handbook:[0-9]-[0-9]*)',\n            '\\\\1 (http://handbook.datalad.org/symbols)',\n            docs)\n    docs = re.sub(\n        r'^([ ]*)\\|\\| REFLOW \\>\\>\\n(.*?)\\<\\< REFLOW \\|\\|',\n        lambda match: textwrap.fill(match.group(2), subsequent_indent=match.group(1)),\n        docs,\n        flags=re.MULTILINE | re.DOTALL)\n    return docs",
                            "ast_type_str": "FunctionDef",
                            "start_lineno": 172,
                            "end_lineno": 224,
                            "parent": null
                        }
                    ],
                    "added_entities": [],
                    "deleted_entities": []
                }
            ]
        },
        {
            "old_file_content": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 et:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Tests for interface doc wranglers.\n\n\"\"\"\n\n__docformat__ = 'restructuredtext'\n\nfrom datalad.interface.base import (\n    alter_interface_docs_for_api,\n    dedent_docstring,\n)\nfrom datalad.tests.utils_pytest import (\n    assert_false,\n    assert_in,\n    assert_not_in,\n)\n\ndemo_doc = \"\"\"\\\n    Bla bla summary\n\n    Generic intro blurb. Ping pong ping pong ping pong ping pong.  Ping pong ping\n    pong ping pong ping pong. Ping pong ping pong ping pong ping pong. Ping pong\n    ping pong ping pong ping pong. Ping pong ping pong ping pong ping pong. Ping\n    pong ping pong ping pong ping pong.\n\n    || CMDLINE >>\n    || REFLOW >>\n    Something for the cmdline only\n    Multiline!\n    << REFLOW ||\n    << CMDLINE ||\n\n    || REFLOW >>\n    a\n    b\n    << REFLOW ||\n    not\n       reflowed\n    || REFLOW >>\n    c\n    << REFLOW ||\n\n    || PYTHON >>\n\n    || REFLOW >>\n    Some Python-only bits\n    Multiline!\n    << REFLOW ||\n\n    << PYTHON ||\n\n    And an example for in-line markup: [PY: just for Python PY] and\n    the other one [CMD: just for the command line CMD]. End of demo.\n\n    Generic appendix. Ding dong ding dong ding dong.  Ding dong ding dong ding\n    dong.  Ding dong ding dong ding dong.  Ding dong ding dong ding dong.  Ding\n    dong ding dong ding dong.\n\n\"\"\"\n\ndemo_paramdoc = \"\"\"\\\n\n    Parameters\n    ----------\n    dataset : Dataset or None, optional\n      something [PY: python only PY] in between [CMD: cmdline only CMD] appended [PY: more python PY]\n      dataset is given, an attempt is made to identify the dataset based\n      Dataset (e.g. a path), or value must be `None`. [Default: None]\n\"\"\"\n\ndemo_argdoc = \"\"\"\\\n    specify the dataset to perform the install operation\n    on. If no dataset is given, an attempt is made to\n    identify the dataset based on the current working\n    directory and/or the `path` given. Constraints: Value\n    must be a Dataset or a valid identifier of a Dataset\n    (e.g. a path), or value must be `None`. [Default:\n    None]\n\"\"\"\n\n\ndef test_dedent():\n    assert_false(dedent_docstring(\"one liner\").endswith(\"\\n\"))\n\n\ndef test_alter_interface_docs_for_api():\n    alt = alter_interface_docs_for_api(demo_doc)\n    alt_l = alt.split('\\n')\n    # de-dented\n    assert_false(alt_l[0].startswith(' '))\n    assert_false(alt_l[-1].startswith(' '))\n    assert_not_in('CMD', alt)\n    assert_not_in('PY', alt)\n    assert_not_in('REFLOW', alt)\n    assert_in('a b', alt)\n    assert_in('not\\n   reflowed', alt)\n    assert_in(\"Some Python-only bits Multiline!\", alt)\n\n    altpd = alter_interface_docs_for_api(demo_paramdoc)\n    assert_in('python', altpd)\n    assert_in('in between', altpd)\n    assert_in('appended', altpd)\n    assert_not_in('cmdline', altpd)\n",
            "new_file_content": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 et:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Tests for interface doc wranglers.\n\n\"\"\"\n\n__docformat__ = 'restructuredtext'\n\nfrom datalad.interface.base import (\n    alter_interface_docs_for_api,\n    dedent_docstring,\n)\nfrom datalad.tests.utils_pytest import (\n    assert_false,\n    assert_in,\n    assert_not_in,\n)\n\ndemo_doc = \"\"\"\\\n    Bla bla summary\n\n    Generic intro blurb. Ping pong ping pong ping pong ping pong.  Ping pong ping\n    pong ping pong ping pong. Ping pong ping pong ping pong ping pong. Ping pong\n    ping pong ping pong ping pong. Ping pong ping pong ping pong ping pong. Ping\n    pong ping pong ping pong ping pong.\n\n    || CMDLINE >>\n    || REFLOW >>\n    Something for the cmdline only\n    Multiline!\n    << REFLOW ||\n    << CMDLINE ||\n\n    || REFLOW >>\n    a\n    b\n    << REFLOW ||\n    not\n       reflowed\n    || REFLOW >>\n    c\n    << REFLOW ||\n\n    || PYTHON >>\n\n    || REFLOW >>\n    Some Python-only bits\n    Multiline!\n    << REFLOW ||\n\n    << PYTHON ||\n\n    And an example for in-line markup: [PY: just for Python PY] and\n    the other one [CMD: just for the command line CMD]. [PY: multiline\n    python-only with [ brackets [] ] PY][CMD: multiline cli-only with [ brackets\n    [] ] CMD]. End of demo.\n\n    Generic appendix. Ding dong ding dong ding dong.  Ding dong ding dong ding\n    dong.  Ding dong ding dong ding dong.  Ding dong ding dong ding dong.  Ding\n    dong ding dong ding dong.\n\n\"\"\"\n\ndemo_paramdoc = \"\"\"\\\n\n    Parameters\n    ----------\n    dataset : Dataset or None, optional\n      something [PY: python only PY] in between [CMD: cmdline only CMD] appended\n      Brackets can also be within and we can deal with [PY: multiline\n      python-only with [ some brackets [] PY] [CMD: multiline cli-only [\n      brackets included [ can we also have || ?] CMD].\n      dataset is given, an attempt is made to identify the dataset based\n      Dataset (e.g. a path), or value must be `None`. [Default: None]\n\"\"\"\n\ndemo_argdoc = \"\"\"\\\n    specify the dataset to perform the install operation\n    on. If no dataset is given, an attempt is made to\n    identify the dataset based on the current working\n    directory and/or the `path` given. Constraints: Value\n    must be a Dataset or a valid identifier of a Dataset\n    (e.g. a path), or value must be `None`. [Default:\n    None]\n\"\"\"\n\n\ndef test_dedent():\n    assert_false(dedent_docstring(\"one liner\").endswith(\"\\n\"))\n\n\ndef test_alter_interface_docs_for_api():\n    alt = alter_interface_docs_for_api(demo_doc)\n    alt_l = alt.split('\\n')\n    # de-dented\n    assert_false(alt_l[0].startswith(' '))\n    assert_false(alt_l[-1].startswith(' '))\n    assert_not_in('CMD', alt)\n    assert_not_in('PY', alt)\n    assert_not_in('REFLOW', alt)\n    assert_in('a b', alt)\n    assert_in('not\\n   reflowed', alt)\n    assert_in(\"Some Python-only bits Multiline!\", alt)\n    assert_in(\"Some Python-only bits\", alt)\n    assert_in(\"just for Python\", alt)\n    assert_not_in(\"just for the command line\", alt)\n    assert_not_in(\"multiline cli-only with [ brackets\\n[] ]\", alt)\n    assert_in(\"multiline\\npython-only with [ brackets [] ]\", alt)\n\n    altpd = alter_interface_docs_for_api(demo_paramdoc)\n    assert_not_in(\"PY\", altpd)\n    assert_not_in(\"CMD\", altpd)\n    assert_in('python', altpd)\n    assert_in('in between', altpd)\n    assert_in('appended', altpd)\n    assert_in(\"multiline\\n  python-only with [ some brackets []\", altpd)\n    assert_not_in('cmdline', altpd)\n    assert_not_in(\"multiline cli-only [\\n  brackets included \"\n                  \"[ can we also have || ?]\", altpd)\n",
            "header": {
                "file": {
                    "path": "datalad/interface/tests/test_docs.py"
                },
                "misc_line": null
            },
            "index_line": {
                "old_commit_hash": "fdf0e872d",
                "new_commit_hash": "7dda2fab5",
                "mode": "100644"
            },
            "is_binary_file": false,
            "binary_line": null,
            "minus_file": {
                "path": "a/datalad/interface/tests/test_docs.py"
            },
            "plus_file": {
                "path": "b/datalad/interface/tests/test_docs.py"
            },
            "hunks": [
                {
                    "descriptor": {
                        "old_range": {
                            "start": 57,
                            "length": 7
                        },
                        "new_range": {
                            "start": 57,
                            "length": 9
                        },
                        "section": "demo_doc = \"\"\"\\"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "    << PYTHON ||",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "    And an example for in-line markup: [PY: just for Python PY] and",
                                "type": "context"
                            },
                            {
                                "content": "    the other one [CMD: just for the command line CMD]. End of demo.",
                                "type": "deleted"
                            },
                            {
                                "content": "    the other one [CMD: just for the command line CMD]. [PY: multiline",
                                "type": "added"
                            },
                            {
                                "content": "    python-only with [ brackets [] ] PY][CMD: multiline cli-only with [ brackets",
                                "type": "added"
                            },
                            {
                                "content": "    [] ] CMD]. End of demo.",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "    Generic appendix. Ding dong ding dong ding dong.  Ding dong ding dong ding",
                                "type": "context"
                            },
                            {
                                "content": "    dong.  Ding dong ding dong ding dong.  Ding dong ding dong ding dong.  Ding",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [],
                    "added_entities": [
                        {
                            "file_name": "datalad/interface/tests/test_docs.py",
                            "type": "statement",
                            "name": "",
                            "content": "demo_doc = \"\"\"\\\n    Bla bla summary\n\n    Generic intro blurb. Ping pong ping pong ping pong ping pong.  Ping pong ping\n    pong ping pong ping pong. Ping pong ping pong ping pong ping pong. Ping pong\n    ping pong ping pong ping pong. Ping pong ping pong ping pong ping pong. Ping\n    pong ping pong ping pong ping pong.\n\n    || CMDLINE >>\n    || REFLOW >>\n    Something for the cmdline only\n    Multiline!\n    << REFLOW ||\n    << CMDLINE ||\n\n    || REFLOW >>\n    a\n    b\n    << REFLOW ||\n    not\n       reflowed\n    || REFLOW >>\n    c\n    << REFLOW ||\n\n    || PYTHON >>\n\n    || REFLOW >>\n    Some Python-only bits\n    Multiline!\n    << REFLOW ||\n\n    << PYTHON ||\n\n    And an example for in-line markup: [PY: just for Python PY] and\n    the other one [CMD: just for the command line CMD]. [PY: multiline\n    python-only with [ brackets [] ] PY][CMD: multiline cli-only with [ brackets\n    [] ] CMD]. End of demo.\n\n    Generic appendix. Ding dong ding dong ding dong.  Ding dong ding dong ding\n    dong.  Ding dong ding dong ding dong.  Ding dong ding dong ding dong.  Ding\n    dong ding dong ding dong.\n\n\"\"\"",
                            "ast_type_str": "Assign",
                            "start_lineno": 25,
                            "end_lineno": 68,
                            "parent": null
                        }
                    ],
                    "deleted_entities": [
                        {
                            "file_name": "datalad/interface/tests/test_docs.py",
                            "type": "statement",
                            "name": "",
                            "content": "demo_doc = \"\"\"\\\n    Bla bla summary\n\n    Generic intro blurb. Ping pong ping pong ping pong ping pong.  Ping pong ping\n    pong ping pong ping pong. Ping pong ping pong ping pong ping pong. Ping pong\n    ping pong ping pong ping pong. Ping pong ping pong ping pong ping pong. Ping\n    pong ping pong ping pong ping pong.\n\n    || CMDLINE >>\n    || REFLOW >>\n    Something for the cmdline only\n    Multiline!\n    << REFLOW ||\n    << CMDLINE ||\n\n    || REFLOW >>\n    a\n    b\n    << REFLOW ||\n    not\n       reflowed\n    || REFLOW >>\n    c\n    << REFLOW ||\n\n    || PYTHON >>\n\n    || REFLOW >>\n    Some Python-only bits\n    Multiline!\n    << REFLOW ||\n\n    << PYTHON ||\n\n    And an example for in-line markup: [PY: just for Python PY] and\n    the other one [CMD: just for the command line CMD]. End of demo.\n\n    Generic appendix. Ding dong ding dong ding dong.  Ding dong ding dong ding\n    dong.  Ding dong ding dong ding dong.  Ding dong ding dong ding dong.  Ding\n    dong ding dong ding dong.\n\n\"\"\"",
                            "ast_type_str": "Assign",
                            "start_lineno": 25,
                            "end_lineno": 66,
                            "parent": null
                        }
                    ]
                },
                {
                    "descriptor": {
                        "old_range": {
                            "start": 70,
                            "length": 7
                        },
                        "new_range": {
                            "start": 72,
                            "length": 10
                        },
                        "section": "demo_paramdoc = \"\"\"\\"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "    Parameters",
                                "type": "context"
                            },
                            {
                                "content": "    ----------",
                                "type": "context"
                            },
                            {
                                "content": "    dataset : Dataset or None, optional",
                                "type": "context"
                            },
                            {
                                "content": "      something [PY: python only PY] in between [CMD: cmdline only CMD] appended [PY: more python PY]",
                                "type": "deleted"
                            },
                            {
                                "content": "      something [PY: python only PY] in between [CMD: cmdline only CMD] appended",
                                "type": "added"
                            },
                            {
                                "content": "      Brackets can also be within and we can deal with [PY: multiline",
                                "type": "added"
                            },
                            {
                                "content": "      python-only with [ some brackets [] PY] [CMD: multiline cli-only [",
                                "type": "added"
                            },
                            {
                                "content": "      brackets included [ can we also have || ?] CMD].",
                                "type": "added"
                            },
                            {
                                "content": "      dataset is given, an attempt is made to identify the dataset based",
                                "type": "context"
                            },
                            {
                                "content": "      Dataset (e.g. a path), or value must be `None`. [Default: None]",
                                "type": "context"
                            },
                            {
                                "content": "\"\"\"",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [],
                    "added_entities": [
                        {
                            "file_name": "datalad/interface/tests/test_docs.py",
                            "type": "statement",
                            "name": "",
                            "content": "demo_paramdoc = \"\"\"\\\n\n    Parameters\n    ----------\n    dataset : Dataset or None, optional\n      something [PY: python only PY] in between [CMD: cmdline only CMD] appended\n      Brackets can also be within and we can deal with [PY: multiline\n      python-only with [ some brackets [] PY] [CMD: multiline cli-only [\n      brackets included [ can we also have || ?] CMD].\n      dataset is given, an attempt is made to identify the dataset based\n      Dataset (e.g. a path), or value must be `None`. [Default: None]\n\"\"\"",
                            "ast_type_str": "Assign",
                            "start_lineno": 70,
                            "end_lineno": 81,
                            "parent": null
                        }
                    ],
                    "deleted_entities": [
                        {
                            "file_name": "datalad/interface/tests/test_docs.py",
                            "type": "statement",
                            "name": "",
                            "content": "demo_paramdoc = \"\"\"\\\n\n    Parameters\n    ----------\n    dataset : Dataset or None, optional\n      something [PY: python only PY] in between [CMD: cmdline only CMD] appended [PY: more python PY]\n      dataset is given, an attempt is made to identify the dataset based\n      Dataset (e.g. a path), or value must be `None`. [Default: None]\n\"\"\"",
                            "ast_type_str": "Assign",
                            "start_lineno": 68,
                            "end_lineno": 76,
                            "parent": null
                        }
                    ]
                },
                {
                    "descriptor": {
                        "old_range": {
                            "start": 102,
                            "length": 9
                        },
                        "new_range": {
                            "start": 107,
                            "length": 19
                        },
                        "section": "def test_alter_interface_docs_for_api():"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "    assert_in('a b', alt)",
                                "type": "context"
                            },
                            {
                                "content": "    assert_in('not\\n   reflowed', alt)",
                                "type": "context"
                            },
                            {
                                "content": "    assert_in(\"Some Python-only bits Multiline!\", alt)",
                                "type": "context"
                            },
                            {
                                "content": "    assert_in(\"Some Python-only bits\", alt)",
                                "type": "added"
                            },
                            {
                                "content": "    assert_in(\"just for Python\", alt)",
                                "type": "added"
                            },
                            {
                                "content": "    assert_not_in(\"just for the command line\", alt)",
                                "type": "added"
                            },
                            {
                                "content": "    assert_not_in(\"multiline cli-only with [ brackets\\n[] ]\", alt)",
                                "type": "added"
                            },
                            {
                                "content": "    assert_in(\"multiline\\npython-only with [ brackets [] ]\", alt)",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "    altpd = alter_interface_docs_for_api(demo_paramdoc)",
                                "type": "context"
                            },
                            {
                                "content": "    assert_not_in(\"PY\", altpd)",
                                "type": "added"
                            },
                            {
                                "content": "    assert_not_in(\"CMD\", altpd)",
                                "type": "added"
                            },
                            {
                                "content": "    assert_in('python', altpd)",
                                "type": "context"
                            },
                            {
                                "content": "    assert_in('in between', altpd)",
                                "type": "context"
                            },
                            {
                                "content": "    assert_in('appended', altpd)",
                                "type": "context"
                            },
                            {
                                "content": "    assert_in(\"multiline\\n  python-only with [ some brackets []\", altpd)",
                                "type": "added"
                            },
                            {
                                "content": "    assert_not_in('cmdline', altpd)",
                                "type": "context"
                            },
                            {
                                "content": "    assert_not_in(\"multiline cli-only [\\n  brackets included \"",
                                "type": "added"
                            },
                            {
                                "content": "                  \"[ can we also have || ?]\", altpd)",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [
                        {
                            "file_name": "datalad/interface/tests/test_docs.py",
                            "type": "function",
                            "name": "test_alter_interface_docs_for_api",
                            "content": "def test_alter_interface_docs_for_api():\n    alt = alter_interface_docs_for_api(demo_doc)\n    alt_l = alt.split('\\n')\n    # de-dented\n    assert_false(alt_l[0].startswith(' '))\n    assert_false(alt_l[-1].startswith(' '))\n    assert_not_in('CMD', alt)\n    assert_not_in('PY', alt)\n    assert_not_in('REFLOW', alt)\n    assert_in('a b', alt)\n    assert_in('not\\n   reflowed', alt)\n    assert_in(\"Some Python-only bits Multiline!\", alt)\n    assert_in(\"Some Python-only bits\", alt)\n    assert_in(\"just for Python\", alt)\n    assert_not_in(\"just for the command line\", alt)\n    assert_not_in(\"multiline cli-only with [ brackets\\n[] ]\", alt)\n    assert_in(\"multiline\\npython-only with [ brackets [] ]\", alt)\n\n    altpd = alter_interface_docs_for_api(demo_paramdoc)\n    assert_not_in(\"PY\", altpd)\n    assert_not_in(\"CMD\", altpd)\n    assert_in('python', altpd)\n    assert_in('in between', altpd)\n    assert_in('appended', altpd)\n    assert_in(\"multiline\\n  python-only with [ some brackets []\", altpd)\n    assert_not_in('cmdline', altpd)\n    assert_not_in(\"multiline cli-only [\\n  brackets included \"\n                  \"[ can we also have || ?]\", altpd)",
                            "ast_type_str": "FunctionDef",
                            "start_lineno": 98,
                            "end_lineno": 125,
                            "parent": null
                        }
                    ],
                    "added_entities": [],
                    "deleted_entities": []
                }
            ]
        }
    ],
    "old_commit_hash": "ad61af78abf25aad6c0d1d569b0da1e2a769ba06^",
    "new_commit_hash": "ad61af78abf25aad6c0d1d569b0da1e2a769ba06",
    "commit_message": "Allow for brackets in PY&CMD doc markups\n\nFixes #7225\n\nInterface specific substitutions did not work with nested brackets (and\nfor some reason also not crossing newlines).\n\nLooks like those regexes were simply meant to not be greedy.",
    "commit_date": "2022-12-14T11:15:17+01:00",
    "metadata": {}
}
{
    "repo_name": "datalad",
    "new_commit_hash": "ad61af78abf25aad6c0d1d569b0da1e2a769ba06",
    "test_file_codes": [
        "from datalad.interface.tests.test_docs import (\n    demo_argdoc,\n    demo_doc,\n    demo_paramdoc,\n)\nfrom datalad.tests.utils_pytest import (\n    assert_false,\n    assert_in,\n    assert_not_in,\n    eq_,\n)\n\nfrom datalad.cli.interface import (\n    alter_interface_docs_for_cmdline,\n    get_cmdline_command_name,\n)\n\n\ndef test_alter_interface_docs_for_cmdline():\n    alt = alter_interface_docs_for_cmdline(demo_doc)\n    alt_l = alt.split('\\n')\n    # de-dented\n    assert_false(alt_l[0].startswith(' '))\n    assert_false(alt_l[-1].startswith(' '))\n    assert_not_in('PY', alt)\n    assert_not_in('CMD', alt)\n    assert_not_in('REFLOW', alt)\n    assert_in('a b', alt)\n    assert_in('not\\n   reflowed', alt)\n    assert_in(\"Something for the cmdline only Multiline!\", alt)\n    assert_not_in(\"Some Python-only bits\", alt)\n    assert_not_in(\"just for Python\", alt)\n    assert_in(\"just for the command line\", alt)\n    assert_in(\"multiline cli-only with [ brackets\\n[] ]\", alt)\n    assert_not_in(\"multiline\\npython-only with [ brackets [] ]\", alt)\n\n    # args\n    altarg = alter_interface_docs_for_cmdline(demo_argdoc)\n    # RST role markup\n    eq_(alter_interface_docs_for_cmdline(':murks:`me and my buddies`'),\n        'me and my buddies')\n    # spread across lines\n    eq_(alter_interface_docs_for_cmdline(':term:`Barbara\\nStreisand`'),\n        'Barbara\\nStreisand')\n    # multiple on one line\n    eq_(alter_interface_docs_for_cmdline(\n        ':term:`one` bla bla :term:`two` bla'),\n        'one bla bla two bla')\n\n    altpd = alter_interface_docs_for_cmdline(demo_paramdoc)\n    assert_not_in(\"PY\", altpd)\n    assert_not_in(\"CMD\", altpd)\n    assert_not_in('python', altpd)\n    assert_not_in(\"python-only with [ some brackets []\", altpd)\n    assert_in('in between', altpd)\n    assert_in('appended', altpd)\n    assert_in('cmdline', altpd)\n    assert_in(\"multiline cli-only [\\n  brackets included \"\n              \"[ can we also have || ?]\", altpd)\n\n\ndef test_name_generation():\n    eq_(\n        get_cmdline_command_name((\"some.module_something\", \"SomeClass\")),\n        \"module-something\")\n    eq_(\n        get_cmdline_command_name((\n            \"some.module_something\",\n            \"SomeClass\",\n            \"override\")),\n        \"override\")\n    eq_(\n        get_cmdline_command_name((\n            \"some.module_something\",\n            \"SomeClass\",\n            \"override\",\n            \"api_ignore\")),\n        \"override\")\n",
        "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 et:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"Tests for interface doc wranglers.\n\n\"\"\"\n\n__docformat__ = 'restructuredtext'\n\nfrom datalad.interface.base import (\n    alter_interface_docs_for_api,\n    dedent_docstring,\n)\nfrom datalad.tests.utils_pytest import (\n    assert_false,\n    assert_in,\n    assert_not_in,\n)\n\ndemo_doc = \"\"\"\\\n    Bla bla summary\n\n    Generic intro blurb. Ping pong ping pong ping pong ping pong.  Ping pong ping\n    pong ping pong ping pong. Ping pong ping pong ping pong ping pong. Ping pong\n    ping pong ping pong ping pong. Ping pong ping pong ping pong ping pong. Ping\n    pong ping pong ping pong ping pong.\n\n    || CMDLINE >>\n    || REFLOW >>\n    Something for the cmdline only\n    Multiline!\n    << REFLOW ||\n    << CMDLINE ||\n\n    || REFLOW >>\n    a\n    b\n    << REFLOW ||\n    not\n       reflowed\n    || REFLOW >>\n    c\n    << REFLOW ||\n\n    || PYTHON >>\n\n    || REFLOW >>\n    Some Python-only bits\n    Multiline!\n    << REFLOW ||\n\n    << PYTHON ||\n\n    And an example for in-line markup: [PY: just for Python PY] and\n    the other one [CMD: just for the command line CMD]. [PY: multiline\n    python-only with [ brackets [] ] PY][CMD: multiline cli-only with [ brackets\n    [] ] CMD]. End of demo.\n\n    Generic appendix. Ding dong ding dong ding dong.  Ding dong ding dong ding\n    dong.  Ding dong ding dong ding dong.  Ding dong ding dong ding dong.  Ding\n    dong ding dong ding dong.\n\n\"\"\"\n\ndemo_paramdoc = \"\"\"\\\n\n    Parameters\n    ----------\n    dataset : Dataset or None, optional\n      something [PY: python only PY] in between [CMD: cmdline only CMD] appended\n      Brackets can also be within and we can deal with [PY: multiline\n      python-only with [ some brackets [] PY] [CMD: multiline cli-only [\n      brackets included [ can we also have || ?] CMD].\n      dataset is given, an attempt is made to identify the dataset based\n      Dataset (e.g. a path), or value must be `None`. [Default: None]\n\"\"\"\n\ndemo_argdoc = \"\"\"\\\n    specify the dataset to perform the install operation\n    on. If no dataset is given, an attempt is made to\n    identify the dataset based on the current working\n    directory and/or the `path` given. Constraints: Value\n    must be a Dataset or a valid identifier of a Dataset\n    (e.g. a path), or value must be `None`. [Default:\n    None]\n\"\"\"\n\n\ndef test_dedent():\n    assert_false(dedent_docstring(\"one liner\").endswith(\"\\n\"))\n\n\ndef test_alter_interface_docs_for_api():\n    alt = alter_interface_docs_for_api(demo_doc)\n    alt_l = alt.split('\\n')\n    # de-dented\n    assert_false(alt_l[0].startswith(' '))\n    assert_false(alt_l[-1].startswith(' '))\n    assert_not_in('CMD', alt)\n    assert_not_in('PY', alt)\n    assert_not_in('REFLOW', alt)\n    assert_in('a b', alt)\n    assert_in('not\\n   reflowed', alt)\n    assert_in(\"Some Python-only bits Multiline!\", alt)\n    assert_in(\"Some Python-only bits\", alt)\n    assert_in(\"just for Python\", alt)\n    assert_not_in(\"just for the command line\", alt)\n    assert_not_in(\"multiline cli-only with [ brackets\\n[] ]\", alt)\n    assert_in(\"multiline\\npython-only with [ brackets [] ]\", alt)\n\n    altpd = alter_interface_docs_for_api(demo_paramdoc)\n    assert_not_in(\"PY\", altpd)\n    assert_not_in(\"CMD\", altpd)\n    assert_in('python', altpd)\n    assert_in('in between', altpd)\n    assert_in('appended', altpd)\n    assert_in(\"multiline\\n  python-only with [ some brackets []\", altpd)\n    assert_not_in('cmdline', altpd)\n    assert_not_in(\"multiline cli-only [\\n  brackets included \"\n                  \"[ can we also have || ?]\", altpd)\n",
        "import pytest\nimport tempfile\nimport os\n\n\n@pytest.fixture\ndef path():\n    with tempfile.TemporaryDirectory() as tmpdir:\n        yield tmpdir\n\n\n@pytest.fixture(autouse=True)\ndef setup_git_config():\n    with tempfile.TemporaryDirectory() as home:\n        old_home = os.environ.get(\"HOME\")\n        os.environ[\"HOME\"] = home\n        os.system('git config --global user.name \"DataLad Tester\"')\n        os.system('git config --global user.email \"test@example.com\"')\n        yield\n        if old_home:\n            os.environ[\"HOME\"] = old_home\n"
    ],
    "test_file_names": [
        "test_1.py",
        "test_2.py",
        "conftest.py"
    ],
    "setup_res_code": 0,
    "setup_res_stdout": "Starting Datalad installation attempts...\nVerifying installation...\nDatalad version: 0.17.9+100.gad61af78a\n✅ Installation successful!\nSuccessfully installed Datalad using Python 3.9\n",
    "setup_res_stderr": "Using CPython 3.9.21\nCreating virtual environment at: .venv\nActivate with: source .venv/bin/activate\nResolved 25 packages in 107ms\nInstalled 25 packages in 4.19s\n + coverage==7.6.10\n + docopt==0.6.2\n + exceptiongroup==1.2.2\n + grabbit==0.2.6\n + importlib-resources==6.4.5\n + iniconfig==2.0.0\n + nibabel==5.3.2\n + num2words==0.5.14\n + numpy==2.0.2\n + packaging==24.2\n + pandas==2.2.3\n + patsy==1.0.1\n + pluggy==1.5.0\n + pybids==0.6.5\n + pytest==8.3.4\n + pytest-cov==6.0.0\n + python-dateutil==2.9.0.post0\n + pytz==2024.2\n + scipy==1.13.1\n + setuptools==75.6.0\n + six==1.17.0\n + tomli==2.2.1\n + typing-extensions==4.12.2\n + tzdata==2024.2\n + zipp==3.21.0\nResolved 72 packages in 4.14s\nPrepared 1 package in 5.05s\nInstalled 59 packages in 337ms\n + annexremote==1.6.6\n + argcomplete==3.5.3\n + backports-tarfile==1.2.0\n + beautifulsoup4==4.12.3\n + boto==2.49.0\n + certifi==2024.12.14\n + cffi==1.17.1\n + chardet==4.0.0\n + charset-normalizer==3.4.1\n + citeproc-py==0.6.0\n + cryptography==44.0.0\n + datalad==0.17.9+100.gad61af78a (from file:///home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_ad61af78abf25aad6c0d1d569b0da1e2a769ba06)\n + distro==1.9.0\n + duecredit==0.10.2\n + exifread==3.0.0\n + fasteners==0.19\n + httpretty==1.1.4\n + humanize==4.11.0\n + idna==3.10\n + importlib-metadata==8.5.0\n + iso8601==2.1.0\n + jaraco-classes==3.4.0\n + jaraco-context==6.0.1\n + jaraco-functools==4.1.0\n + jeepney==0.8.0\n + keyring==25.6.0\n + keyrings-alt==5.0.2\n + looseversion==1.3.0\n + lxml==5.3.0\n + more-itertools==10.5.0\n + msgpack==1.1.0\n + multidict==6.1.0\n + mutagen==1.47.0\n + mypy==0.991\n + mypy-extensions==1.0.0\n + patool==1.15.0\n + pillow==11.0.0\n + platformdirs==4.3.6\n + propcache==0.2.1\n + pycparser==2.22\n + pyperclip==1.9.0\n + pytest-fail-slow==0.6.0\n + python-gitlab==5.3.0\n + python-xmp-toolkit==2.0.2\n + pyyaml==6.0.2\n + requests==2.32.3\n + requests-ftp==0.3.1\n + requests-toolbelt==1.0.0\n + secretstorage==3.3.3\n + simplejson==3.19.3\n + soupsieve==2.6\n + tqdm==4.67.1\n + types-python-dateutil==2.9.0.20241206\n + types-requests==2.32.0.20241016\n + urllib3==2.3.0\n + vcrpy==4.3.0\n + whoosh==2.7.4\n + wrapt==1.17.0\n + yarl==1.18.3\nResolved 104 packages in 307ms\nInstalled 32 packages in 508ms\n + alabaster==0.7.16\n + asv==0.6.4\n + asv-runner==0.2.1\n + babel==2.16.0\n + build==1.2.2.post1\n + distlib==0.3.9\n + docutils==0.21.2\n + execnet==2.1.1\n + filelock==3.16.1\n + gprof2dot==2024.6.6\n + imagesize==1.4.1\n + jinja2==3.1.5\n + json5==0.10.0\n + markupsafe==3.0.2\n + psutil==6.1.1\n + pygments==2.18.0\n + pympler==1.1\n + pypandoc==1.14\n + pyproject-hooks==1.2.0\n + pytest-xdist==3.6.1\n + snowballstemmer==2.2.0\n + sphinx==7.4.7\n + sphinx-rtd-theme==3.0.2\n + sphinxcontrib-applehelp==2.0.0\n + sphinxcontrib-devhelp==2.0.0\n + sphinxcontrib-htmlhelp==2.1.0\n + sphinxcontrib-jquery==4.1\n + sphinxcontrib-jsmath==1.0.1\n + sphinxcontrib-qthelp==2.0.0\n + sphinxcontrib-serializinghtml==2.0.0\n + tabulate==0.9.0\n + virtualenv==20.28.0\n",
    "new_commit_res_code": 0,
    "new_commit_res_stdout": "============================= test session starts ==============================\nplatform linux -- Python 3.9.21, pytest-8.3.4, pluggy-1.5.0\nrootdir: /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_ad61af78abf25aad6c0d1d569b0da1e2a769ba06\nconfigfile: tox.ini\nplugins: fail-slow-0.6.0, xdist-3.6.1, cov-6.0.0\ncollected 4 items\n\nr2e_tests/test_1.py ..                                                   [ 50%]\nr2e_tests/test_2.py ..                                                   [100%]\n\n==================================== PASSES ====================================\n=========================== short test summary info ============================\nPASSED r2e_tests/test_1.py::test_alter_interface_docs_for_cmdline\nPASSED r2e_tests/test_1.py::test_name_generation\nPASSED r2e_tests/test_2.py::test_dedent\nPASSED r2e_tests/test_2.py::test_alter_interface_docs_for_api\n============================== 4 passed in 0.65s ===============================\n",
    "new_commit_res_stderr": "",
    "old_commit_res_code": 1,
    "old_commit_res_stdout": "============================= test session starts ==============================\nplatform linux -- Python 3.9.21, pytest-8.3.4, pluggy-1.5.0\nrootdir: /home/gcpuser/buckets/local_repoeval_bucket/repos/datalad_ad61af78abf25aad6c0d1d569b0da1e2a769ba06\nconfigfile: tox.ini\nplugins: fail-slow-0.6.0, xdist-3.6.1, cov-6.0.0\ncollected 4 items\n\nr2e_tests/test_1.py F.                                                   [ 50%]\nr2e_tests/test_2.py .F                                                   [100%]\n\n=================================== FAILURES ===================================\n____________________ test_alter_interface_docs_for_cmdline _____________________\n\n    def test_alter_interface_docs_for_cmdline():\n        alt = alter_interface_docs_for_cmdline(demo_doc)\n        alt_l = alt.split('\\n')\n        # de-dented\n        assert_false(alt_l[0].startswith(' '))\n        assert_false(alt_l[-1].startswith(' '))\n        assert_not_in('PY', alt)\n        assert_not_in('CMD', alt)\n        assert_not_in('REFLOW', alt)\n        assert_in('a b', alt)\n        assert_in('not\\n   reflowed', alt)\n        assert_in(\"Something for the cmdline only Multiline!\", alt)\n        assert_not_in(\"Some Python-only bits\", alt)\n        assert_not_in(\"just for Python\", alt)\n        assert_in(\"just for the command line\", alt)\n>       assert_in(\"multiline cli-only with [ brackets\\n[] ]\", alt)\n\nr2e_tests/test_1.py:34: \n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \n\nfirst = 'multiline cli-only with [ brackets\\n[] ]'\nsecond = 'Bla bla summary\\n\\nGeneric intro blurb. Ping pong ping pong ping pong ping pong.  Ping pong ping\\npong ping pong ping... dong ding\\ndong.  Ding dong ding dong ding dong.  Ding dong ding dong ding dong.  Ding\\ndong ding dong ding dong.\\n\\n'\nmsg = None\n\n    def assert_in(first, second, msg=None):\n        if msg is None:\n>           assert first in second\nE           AssertionError: assert 'multiline cli-only with [ brackets\\n[] ]' in 'Bla bla summary\\n\\nGeneric intro blurb. Ping pong ping pong ping pong ping pong.  Ping pong ping\\npong ping pong ping... dong ding\\ndong.  Ding dong ding dong ding dong.  Ding dong ding dong ding dong.  Ding\\ndong ding dong ding dong.\\n\\n'\n\ndatalad/tests/utils_pytest.py:131: AssertionError\n______________________ test_alter_interface_docs_for_api _______________________\n\n    def test_alter_interface_docs_for_api():\n        alt = alter_interface_docs_for_api(demo_doc)\n        alt_l = alt.split('\\n')\n        # de-dented\n        assert_false(alt_l[0].startswith(' '))\n        assert_false(alt_l[-1].startswith(' '))\n>       assert_not_in('CMD', alt)\n\nr2e_tests/test_2.py:104: \n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \n\nfirst = 'CMD'\nsecond = 'Bla bla summary\\n\\nGeneric intro blurb. Ping pong ping pong ping pong ping pong.  Ping pong ping\\npong ping pong ping... dong ding\\ndong.  Ding dong ding dong ding dong.  Ding dong ding dong ding dong.  Ding\\ndong ding dong ding dong.\\n\\n'\nmsg = None\n\n    def assert_not_in(first, second, msg=None):\n        if msg is None:\n>           assert first not in second\nE           AssertionError: assert 'CMD' not in 'Bla bla sum...ng dong.\\n\\n'\nE             \nE             'CMD' is contained here:\nE                [] ] PY][CMD: multiline cli-only with [ brackets\nE             ?           +++\nE               [] ] CMD]. End of demo.\nE               \nE               Generic appendix. Ding dong ding dong ding dong.  Ding dong ding dong ding...\nE             \nE             ...Full output truncated (3 lines hidden), use '-vv' to show\n\ndatalad/tests/utils_pytest.py:175: AssertionError\n==================================== PASSES ====================================\n=========================== short test summary info ============================\nPASSED r2e_tests/test_1.py::test_name_generation\nPASSED r2e_tests/test_2.py::test_dedent\nFAILED r2e_tests/test_1.py::test_alter_interface_docs_for_cmdline - Assertion...\nFAILED r2e_tests/test_2.py::test_alter_interface_docs_for_api - AssertionErro...\n========================= 2 failed, 2 passed in 0.54s ==========================\n",
    "old_commit_res_stderr": ""
}
changelog.d/pr-7226.md datalad/cli/interface.py datalad/cli/tests/test_interface.py datalad/interface/base.py datalad/interface/tests/test_docs.py
We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
[ISSUE]
**Title:** Interface substitutions fail with nested brackets in PY&CMD doc markups

**Description:**
When using nested brackets or multiline content within PY and CMD markup in documentation, the substitution process does not properly remove the markup tags. This results in leftover `PY` and `CMD` tags in the generated documentation, causing the output to be incorrect.

**Example Buggy Code:**
```python
def alter_interface_docs_for_cmdline(docs):
    # Original substitution regex was greedy and failed with nested brackets
    docs = re.sub(r'\[CMD:\s[^\[\]]*\sCMD\]', '', docs, flags=re.MULTILINE)
    docs = re.sub(r'\[PY:\s([^\[\]]*)\sPY\]', lambda match: match.group(1), docs, flags=re.MULTILINE)
    return docs

# Example documentation string with nested brackets
demo_doc = """
Some introductory text.

[PY: multiline python-only with [ brackets [] ] PY][CMD: multiline cli-only with [ brackets [] ] CMD]. End of demo.
"""
```

**Expected Behavior:**
The substitution process should correctly remove the `[PY: ... PY]` and `[CMD: ... CMD]` tags, even when nested brackets or newlines are present, leaving only the inner content. For example, the above `demo_doc` should be transformed to:

```
Some introductory text.

multiline python-only with [ brackets [] ] multiline cli-only with [ brackets [] ]. End of demo.
```

**Actual Behavior:**
The markup tags are not fully removed, leaving `PY` and `CMD` in the output. As a result, the transformed documentation appears as:

```
Some introductory text.

multiline python-only with [ brackets [] ] PY][CMD: multiline cli-only with [ brackets [] ] CMD]. End of demo.
```

This leads to improperly formatted documentation where the `PY` and `CMD` tags remain visible.
[/ISSUE]
--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.

--- BEGIN FILE ---
```
### datalad/interface/base.py
# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-
# ex: set sts=4 ts=4 sw=4 et:
# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
#
#   See COPYING file distributed along with the datalad package for the
#   copyright and license terms.
#
# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
"""High-level interface generation

"""

__docformat__ = 'restructuredtext'

import logging

lgr = logging.getLogger('datalad.interface.base')

import os
import re
import textwrap
import warnings
from abc import (
    ABC,
    abstractmethod,
)
from collections import OrderedDict
from importlib import import_module

import datalad
from datalad.distribution.dataset import (
    Dataset,
    resolve_path,
)
from datalad.interface.common_opts import eval_params
from datalad.support.exceptions import CapturedException

default_logchannels = {
    '': 'debug',
    'ok': 'debug',
    'notneeded': 'debug',
    'impossible': 'warning',
    'error': 'error',
}


def get_api_name(intfspec):
    """Given an interface specification return an API name for it"""
    if len(intfspec) > 3:
        name = intfspec[3]
    else:
        name = intfspec[0].split('.')[-1]
    return name


def get_interface_groups(include_plugins=False):
    """Return a list of command groups.

    Returns
    -------
    A list of tuples with the form (GROUP_NAME, GROUP_DESCRIPTION, COMMANDS).
    """
    if include_plugins:
        warnings.warn("Plugins are no longer supported.", DeprecationWarning)

    from .. import interface as _interfaces

    grps = []
    # auto detect all available interfaces and generate a function-based
    # API from them
    for _item in _interfaces.__dict__:
        if not _item.startswith('_group_'):
            continue
        grp_name = _item[7:]
        grp = getattr(_interfaces, _item)
        grps.append((grp_name,) + grp)
    return grps


def get_cmd_summaries(descriptions, groups, width=79):
    """Return summaries for the commands in `groups`.

    Parameters
    ----------
    descriptions : dict
        A map of group names to summaries.
    groups : list of tuples
        A list of groups and commands in the form described by
        `get_interface_groups`.
    width : int, optional
        The maximum width of each line in the summary text.

    Returns
    -------
    A list with a formatted entry for each command. The first command of each
    group is preceded by an entry describing the group.
    """
    cmd_summary = []
    for grp in sorted(groups, key=lambda x: x[0]):
        grp_descr = grp[1]
        grp_cmds = descriptions[grp[0]]

        cmd_summary.append('\n*%s*\n' % (grp_descr,))
        for cd in grp_cmds:
            cmd_summary.append('  %s\n%s'
                               % ((cd[0],
                                   textwrap.fill(
                                       cd[1].rstrip(' .'),
                                       width - 5,
                                       initial_indent=' ' * 6,
                                       subsequent_indent=' ' * 6))))
    return cmd_summary


def load_interface(spec):
    """Load and return the class for `spec`.

    Parameters
    ----------
    spec : tuple
        For a standard interface, the first item is the datalad source module
        and the second object name for the interface.

    Returns
    -------
    The interface class or, if importing the module fails, None.
    """
    lgr.log(5, "Importing module %s ", spec[0])
    try:
        mod = import_module(spec[0], package='datalad')
    except Exception as e:
        ce = CapturedException(e)
        lgr.error("Internal error, cannot import interface '%s': %s",
                  spec[0], ce)
        intf = None
    else:
        intf = getattr(mod, spec[1])
    return intf


def get_cmd_doc(interface):
    """Return the documentation for the command defined by `interface`.

    Parameters
    ----------
    interface : subclass of Interface
    """
    intf_doc = '' if interface.__doc__ is None else interface.__doc__.strip()
    if hasattr(interface, '_docs_'):
        # expand docs
        intf_doc = intf_doc.format(**interface._docs_)
    return intf_doc


def dedent_docstring(text):
    """Remove uniform indentation from a multiline docstring"""
    # Problem is that first line might often have no offset, so might
    # need to be ignored from dedent call
    if text is None:
        return None
    if not text.startswith(' '):
        lines = text.split('\n')
        if len(lines) == 1:
            # single line, no indentation, nothing to do
            return text
        text2 = '\n'.join(lines[1:])
        return lines[0] + "\n" + textwrap.dedent(text2)
    else:
        return textwrap.dedent(text)


def alter_interface_docs_for_api(docs):
    """Apply modifications to interface docstrings for Python API use."""
    # central place to alter the impression of docstrings,
    # like removing cmdline specific sections
    if not docs:
        return docs
    docs = dedent_docstring(docs)
    # clean cmdline sections
    docs = re.sub(
        r'\|\| CMDLINE \>\>.*?\<\< CMDLINE \|\|',
        '',
        docs,
        flags=re.MULTILINE | re.DOTALL)
    # clean cmdline in-line bits
    docs = re.sub(
        r'\[CMD:\s[^\[\]]*\sCMD\]',
        '',
        docs,
        flags=re.MULTILINE | re.DOTALL)
    docs = re.sub(
        r'\[PY:\s([^\[\]]*)\sPY\]',
        lambda match: match.group(1),
        docs,
        flags=re.MULTILINE)
    # select only the python alternative from argument specifications
    docs = re.sub(
        r'``([a-zA-Z0-9_,.]+)\|\|([a-zA-Z0-9-,.]+)``',
        lambda match: f'``{match.group(1)}``',
        docs)
    docs = re.sub(
        r'\|\| PYTHON \>\>(.*?)\<\< PYTHON \|\|',
        lambda match: match.group(1),
        docs,
        flags=re.MULTILINE | re.DOTALL)
    if 'DATALAD_SPHINX_RUN' not in os.environ:
        # remove :role:`...` RST markup for cmdline docs
        docs = re.sub(
            r':\S+:`[^`]*`[\\]*',
            lambda match: ':'.join(match.group(0).split(':')[2:]).strip('`\\'),
            docs,
            flags=re.MULTILINE | re.DOTALL)
        # make the handbook doc references more accessible
        # the URL is a redirect configured at readthedocs
        docs = re.sub(
            r'(handbook:[0-9]-[0-9]*)',
            '\\1 (http://handbook.datalad.org/symbols)',
            docs)
    docs = re.sub(
        r'^([ ]*)\|\| REFLOW \>\>\n(.*?)\<\< REFLOW \|\|',
        lambda match: textwrap.fill(match.group(2), subsequent_indent=match.group(1)),
        docs,
        flags=re.MULTILINE | re.DOTALL)
    return docs


def is_api_arg(arg):
    """Return True if argument is our API argument or self or used for internal
    purposes
    """
    return arg != 'self' and not arg.startswith('_')


def update_docstring_with_parameters(func, params, prefix=None, suffix=None,
                                     add_args=None):
    """Generate a useful docstring from a parameter spec

    Amends any existing docstring of a callable with a textual
    description of its parameters. The Parameter spec needs to match
    the number and names of the callables arguments.
    """
    from datalad.utils import getargspec

    # get the signature
    args, varargs, varkw, defaults = getargspec(func, include_kwonlyargs=True)
    defaults = defaults or tuple()
    if add_args:
        add_argnames = sorted(add_args.keys())
        args.extend(add_argnames)
        defaults = defaults + tuple(add_args[k] for k in add_argnames)
    ndefaults = len(defaults)
    # start documentation with what the callable brings with it
    doc = prefix if prefix else u''
    if len(args) > 1:
        if len(doc):
            if not doc.endswith('\n'):
                doc += '\n'
            doc += '\n'
        doc += "Parameters\n----------\n"
        for i, arg in enumerate(args):
            if not is_api_arg(arg):
                continue
            # we need a parameter spec for each argument
            if not arg in params:
                raise ValueError("function has argument '%s' not described as a parameter" % arg)
            param = params[arg]
            # validate the default -- to make sure that the parameter description is
            # somewhat OK
            defaults_idx = ndefaults - len(args) + i
            if defaults_idx >= 0:
                if param.constraints is not None:
                    param.constraints(defaults[defaults_idx])
            orig_docs = param._doc
            param._doc = alter_interface_docs_for_api(param._doc)
            doc += param.get_autodoc(
                arg,
                default=defaults[defaults_idx] if defaults_idx >= 0 else None,
                has_default=defaults_idx >= 0)
            param._doc = orig_docs
            doc += '\n'
    doc += suffix if suffix else u""
    # assign the amended docs
    func.__doc__ = doc
    return func


# TODO should export code_field and indicator, rather than have modes
# TODO this should be a doc helper
def build_example(example, api='python'):
    """Build a code example.

    Take a dict from a classes _example_ specification (list of dicts) and
    build a string with an api or cmd example (for use in cmd help or
    docstring).

    Parameters
    ----------
    api : {'python', 'cmdline'}
        If 'python', build Python example for docstring. If 'cmdline', build
        cmd example.

    Returns
    -------
    ex : str
        Concatenated examples for the given class.
    """
    if api == 'python' :
        code_field='code_py'
        indicator='>'
    elif api == 'cmdline':
        code_field='code_cmd'
        indicator='%'
    else:
        raise ValueError("unknown API selection: {}".format(api))
    if code_field not in example:
        # only show an example if it exist for the API
        return ''
    description = textwrap.fill(example.get('text'))
    # this indent the code snippet to get it properly rendered as code
    # we are not using textwrap.fill(), because it would not acknowledge
    # any meaningful structure/formatting of code snippets. Instead, we
    # maintain line content as is.
    code = dedent_docstring(example.get(code_field))
    needs_indicator = not code.startswith(indicator)
    code = textwrap.indent(code, ' ' * (5 if needs_indicator else 3)).lstrip()

    ex = """{}::\n\n   {}{}\n\n""".format(
        description,
        # disable automatic prefixing, if the example already has one
        # this enables providing more complex examples without having
        # to infer its inner structure
        '{} '.format(indicator)
        if needs_indicator
        # maintain spacing to avoid undesired relative indentation
        else '',
        code)

    return ex


def update_docstring_with_examples(cls_doc, ex):
    """Update a commands docstring with examples.

    Take _examples_ of a command, build the Python examples, and append
    them to the docstring.

    Parameters
    ----------
    cls_doc: str
      docstring
    ex: list
      list of dicts with examples
    """
    from textwrap import indent
    if len(cls_doc):
        cls_doc += "\n"
    cls_doc += "    Examples\n    --------\n"
    # loop though provided examples
    for example in ex:
        cls_doc += indent(build_example(example, api='python'), ' '*4)

    return cls_doc


def build_doc(cls, **kwargs):
    """Decorator to build docstrings for datalad commands

    It's intended to decorate the class, the __call__-method of which is the
    actual command. It expects that __call__-method to be decorated by
    eval_results.

    Note that values for any `eval_params` keys in `cls._params_` are
    ignored.  This means one class may extend another's `_params_`
    without worrying about filtering out `eval_params`.

    Parameters
    ----------
    cls: Interface
      DataLad command implementation
    """
    if datalad.in_librarymode():
        lgr.debug("Not assembling DataLad API docs in libary-mode")
        return cls

    # Note, that this is a class decorator, which is executed only once when the
    # class is imported. It builds the docstring for the class' __call__ method
    # and returns the original class.
    #
    # This is because a decorator for the actual function would not be able to
    # behave like this. To build the docstring we need to access the attribute
    # _params of the class. From within a function decorator we cannot do this
    # during import time, since the class is being built in this very moment and
    # is not yet available in the module. And if we do it from within the part
    # of a function decorator, that is executed when the function is called, we
    # would need to actually call the command once in order to build this
    # docstring.

    lgr.debug("Building doc for %s", cls)

    cls_doc = cls.__doc__
    if hasattr(cls, '_docs_'):
        # expand docs
        cls_doc = cls_doc.format(**cls._docs_)
    # get examples
    ex = getattr(cls, '_examples_', [])
    if ex:
        cls_doc = update_docstring_with_examples(cls_doc, ex)

    call_doc = None
    # suffix for update_docstring_with_parameters:
    if cls.__call__.__doc__:
        call_doc = cls.__call__.__doc__

    # build standard doc and insert eval_doc
    spec = getattr(cls, '_params_', dict())


    # update class attributes that may override defaults
    if not _has_eval_results_call(cls):
        add_args = None
    else:
        # defaults for all common parameters are guaranteed to be available
        # from the class
        add_args = {k: getattr(cls, k) for k in eval_params}

    # ATTN: An important consequence of this update() call is that it
    # fulfills the docstring's promise of overriding any existing
    # values for eval_params keys in _params_.
    #

    # get docs for eval_results parameters:
    spec.update(eval_params)

    update_docstring_with_parameters(
        cls.__call__, spec,
        prefix=alter_interface_docs_for_api(cls_doc),
        suffix=alter_interface_docs_for_api(call_doc),
        add_args=add_args
    )

    if hasattr(cls.__call__, '_dataset_method'):
        cls.__call__._dataset_method.__doc__ = cls.__call__.__doc__

    # return original
    return cls


class Interface(ABC):
    '''Abstract base class for DataLad command implementations

    Any DataLad command implementation must be derived from this class. The
    code snippet below shows a complete sketch of a Python class with such an
    implementation.

    Importantly, no instances of command classes will created. Instead the main
    entry point is a static ``__call__()`` method, which must be implemented
    for any command. It is incorporated as a function in :mod:`datalad.api`, by
    default under the name of the file the implementation resides (e.g.,
    ``command`` for a ``command.py`` file).  Therefore the file should have a
    name that is a syntax-compliant function name. The default naming rule can
    be overwritten with an explicit alternative name (see
    :func:`datalad.interface.base.get_api_name`).

    For commands implementing functionality that is operating on DataLad
    datasets, a command can be also be bound to the
    :class:`~datalad.distribution.dataset.Dataset` class as a method using
    the ``@datasetmethod`` decorator, under the specified name.

    Any ``__call__()`` implementation should be decorated with
    :func:`datalad.interface.utils.eval_results`. This adds support for
    standard result processing, and a range of common command parameters that
    do not need to be manually added to the signature of ``__call__()``. Any
    implementation decorated in this way should be implemented as a generator,
    and ``yield`` :ref:`result records <chap_design_result_records>`.

    Any argument or keyword argument that appears in the signature of
    ``__call__()`` must have a matching item in :attr:`Interface._params_`.
    The dictionary maps argument names to
    :class:`datalad.support.param.Parameter` specifications. The specification
    contain CLI argument declarations, value constraint and data type
    conversation specifications, documentation, and optional
    ``argparse``-specific arguments for CLI parser construction.

    The class decorator :func:`datalad.interface.base.build_doc` inspects an
    :class:`Interface` implementation, and builds a standard docstring from
    various sources of structured information within the class (also see
    below). The documentation is automatically tuned differently, depending on
    the target API (Python vs CLI).

    .. code:: python

        @build_doc
        class ExampleCommand(Interface):
            """SHORT DESCRIPTION

            LONG DESCRIPTION
            ...
            """

            # COMMAND PARAMETER DEFINITIONS
            _params_ = dict(
                example=Parameter(
                    args=("--example",),
                    doc="""Parameter description....""",
                    constraints=...),
                ...
                )
            )

            # RESULT PARAMETER OVERRIDES
            return_type= 'list'
            ...

            # USAGE EXAMPLES
            _examples_ = [
                dict(text="Example description...",
                     code_py="Example Python code...",
                     code_cmd="Example shell code ..."),
                ...
            ]

            @staticmethod
            @datasetmethod(name='example_command')
            @eval_results
            def __call__(example=None, ...):
                ...

                yield dict(...)

    The basic implementation setup described above can be customized for
    individual commands in various way that alter the behavior and
    presentation of a specific command. The following overview uses
    the code comment markers in the above snippet to illustrate where
    in the class implementation these adjustments can be made.

    (SHORT/LONG) DESCRIPTION

    ``Interface.short_description`` can be defined to provide an
    explicit short description to be used in documentation and help output,
    replacing the auto-generated extract from the first line of the full
    description.

    COMMAND PARAMETER DEFINITIONS

    When a parameter specification declares ``Parameter(args=tuple(), ...)``,
    i.e. no arguments specified, it will be ignored by the CLI. Likewise, any
    ``Parameter`` specification for which :func:`is_api_arg` returns ``False``
    will also be ignored by the CLI. Additionally, any such parameter will
    not be added to the parameter description list in the Python docstring.

    RESULT PARAMETER OVERRIDES

    The :func:`datalad.interface.utils.eval_results` decorator automatically
    add a range of additional arguments to a command, which are defined in
    :py:data:`datalad.interface.common_opts.eval_params`. For any such
    parameter an Interface implementation can define an interface-specific
    default value, by declaring a class member with the respective parameter
    name and the desired default as its assigned value. This feature can be
    used to tune the default command behavior, for example, with respect to the
    default result rendering style, or its error behavior.

    In addition to the common parameters of the Python API, an additional
    ``Interface.result_renderer_cmdline`` can be defined, in order to
    instruct the CLI to prefer the specified alternative result renderer
    over an ``Interface.result_renderer`` specification.

    USAGE EXAMPLES

    Any number of usage examples can be described in an ``_examples_`` list
    class attribute. Such an example contains a description, and code examples
    for Python and CLI.
    '''
    _params_ = {}

    @abstractmethod
    def __call__():
        """Must be implemented by any command"""

    # https://github.com/datalad/datalad/issues/6376
    @classmethod
    def get_refds_path(cls, dataset):
        """Return a resolved reference dataset path from a `dataset` argument

        .. deprecated:: 0.16
           Use ``require_dataset()`` instead.
        """
        # theoretically a dataset could come in as a relative path -> resolve
        if dataset is None:
            return dataset
        refds_path = dataset.path if isinstance(dataset, Dataset) \
            else Dataset(dataset).path
        if refds_path:
            refds_path = str(resolve_path(refds_path))
        return refds_path


# pull all defaults from all eval_results() related parameters and assign them
# as attributes to the class, which then becomes the one place to query for
# default and potential overrides
for k, p in eval_params.items():
    setattr(Interface,
            # name is always given
            k,
            # but there may be no default (rather unlikely, though)
            p.cmd_kwargs.get('default', None))


def get_allargs_as_kwargs(call, args, kwargs):
    """Generate a kwargs dict from a call signature and ``*args``, ``**kwargs``

    Basically resolving the argnames for all positional arguments, and
    resolving the defaults for all kwargs that are not given in a kwargs
    dict
    """
    from datalad.utils import getargspec
    argspec = getargspec(call, include_kwonlyargs=True)
    defaults = argspec.defaults
    nargs = len(argspec.args)
    defaults = defaults or []  # ensure it is a list and not None
    assert (nargs >= len(defaults))
    # map any args to their name
    argmap = list(zip(argspec.args[:len(args)], args))
    kwargs_ = OrderedDict(argmap)
    # map defaults of kwargs to their names (update below)
    for k, v in zip(argspec.args[-len(defaults):], defaults):
        if k not in kwargs_:
            kwargs_[k] = v
    # update with provided kwarg args
    kwargs_.update(kwargs)
    # XXX we cannot assert the following, because our own highlevel
    # API commands support more kwargs than what is discoverable
    # from their signature...
    #assert (nargs == len(kwargs_))
    return kwargs_


# Only needed to support command implementations before the introduction
# of @eval_results
def _has_eval_results_call(cls):
    """Return True if cls has a __call__ decorated with @eval_results
    """
    return getattr(getattr(cls, '__call__', None), '_eval_results', False)


### datalad/cli/interface.py
"""Utilities and definitions for DataLad command interfaces"""

# TODO this should be a dochelper
from datalad.interface.base import dedent_docstring

# Some known extensions and their commands to suggest whenever lookup fails
_known_extension_commands = {
    'datalad-container': (
        'containers-list', 'containers-remove', 'containers-add',
        'containers-run'),
    'datalad-crawler': ('crawl', 'crawl-init'),
    'datalad-deprecated': ('ls',),
    'datalad-neuroimaging': ('bids2scidata',)
}

_deprecated_commands = {
    'add': "save",
    'uninstall': 'drop',
}


def get_cmd_ex(interface):
    """Return the examples for the command defined by 'interface'.

    Parameters
    ----------
    interface : subclass of Interface
    """
    from datalad.interface.base import build_example
    intf_ex = "\n\n*Examples*\n\n"
    for example in interface._examples_:
        intf_ex += build_example(example, api='cmdline')
    return intf_ex


def get_cmdline_command_name(intfspec):
    """Given an interface specification return a cmdline command name"""
    if len(intfspec) > 2:
        name = intfspec[2]
    else:
        name = intfspec[0].split('.')[-1].replace('_', '-')
    return name


def alter_interface_docs_for_cmdline(docs):
    """Apply modifications to interface docstrings for cmdline doc use."""
    # central place to alter the impression of docstrings,
    # like removing Python API specific sections, and argument markup
    if not docs:
        return docs
    import re
    import textwrap

    docs = dedent_docstring(docs)
    # clean cmdline sections
    docs = re.sub(
        r'\|\| PYTHON \>\>.*?\<\< PYTHON \|\|',
        '',
        docs,
        flags=re.MULTILINE | re.DOTALL)
    # clean cmdline in-line bits
    docs = re.sub(
        r'\[PY:\s[^\[\]]*\sPY\]',
        '',
        docs,
        flags=re.MULTILINE | re.DOTALL)
    docs = re.sub(
        r'\[CMD:\s([^\[\]]*)\sCMD\]',
        lambda match: match.group(1),
        docs,
        flags=re.MULTILINE)
    docs = re.sub(
        r'\|\| CMDLINE \>\>(.*?)\<\< CMDLINE \|\|',
        lambda match: match.group(1),
        docs,
        flags=re.MULTILINE | re.DOTALL)
    # remove :role:`...` RST markup for cmdline docs
    docs = re.sub(
        r':\S+:`[^`]*`[\\]*',
        lambda match: ':'.join(match.group(0).split(':')[2:]).strip('`\\'),
        docs,
        flags=re.MULTILINE | re.DOTALL)
    # make the handbook doc references more accessible
    # the URL is a redirect configured at readthedocs
    docs = re.sub(
        r'(handbook:[0-9]-[0-9]*)',
        '\\1 (http://handbook.datalad.org/symbols)',
        docs)
    # remove None constraint. In general, `None` on the cmdline means don't
    # give option at all, but specifying `None` explicitly is practically
    # impossible
    docs = re.sub(
        r',\sor\svalue\smust\sbe\s`None`',
        '',
        docs,
        flags=re.MULTILINE | re.DOTALL)
    # capitalize variables and remove backticks to uniformize with
    # argparse output
    docs = re.sub(
        r'([^`]+)`([a-zA-Z0-9_]+)`([^`]+)',
        lambda match: f'{match.group(1)}{match.group(2).upper()}{match.group(3)}',
        docs)
    # select only the cmdline alternative from argument specifications
    docs = re.sub(
        r'``([a-zA-Z0-9_,.]+)\|\|([a-zA-Z0-9-,.]+)``',
        lambda match: f'``{match.group(2)}``',
        docs)
    # clean up sphinx API refs
    docs = re.sub(
        r'\~datalad\.api\.\S*',
        lambda match: "`{0}`".format(match.group(0)[13:]),
        docs)
    # dedicated support for version markup
    docs = docs.replace('.. versionadded::', 'New in version')
    docs = docs.replace('.. versionchanged::', 'Changed in version')
    docs = docs.replace('.. deprecated::', 'Deprecated in version')
    # Remove RST paragraph markup
    docs = re.sub(
        r'^.. \S+::',
        lambda match: match.group(0)[3:-2].upper(),
        docs,
        flags=re.MULTILINE)
    docs = re.sub(
        r'^([ ]*)\|\| REFLOW \>\>\n(.*?)\<\< REFLOW \|\|',
        lambda match: textwrap.fill(match.group(2), subsequent_indent=match.group(1)),
        docs,
        flags=re.MULTILINE | re.DOTALL)
    return docs

```
--- END FILE ---

Please first localize the bug based on the issue statement, and then generate *SEARCH/REPLACE* edits to fix the issue.

Every *SEARCH/REPLACE* edit must use this format:
1. The file path
2. The start of search block: <<<<<<< SEARCH
3. A contiguous chunk of lines to search for in the existing source code
4. The dividing line: =======
5. The lines to replace into the source code
6. The end of the replace block: >>>>>>> REPLACE

Here is an example:

```python
### mathweb/flask/app.py
<<<<<<< SEARCH
from flask import Flask
=======
import math
from flask import Flask
>>>>>>> REPLACE
```
</solution>

Please note that the *SEARCH/REPLACE* edit REQUIRES PROPER INDENTATION. If you would like to add the line '        print(x)', you must fully write that out, with all those spaces before the code!
Wrap each *SEARCH/REPLACE* edit in a code block as shown in the example above. If you have multiple *SEARCH/REPLACE* edits, use a separate code block for each one.
[ISSUE]
**Title:** Interface substitutions fail with nested brackets in PY&CMD doc markups

**Description:**
When using nested brackets or multiline content within PY and CMD markup in documentation, the substitution process does not properly remove the markup tags. This results in leftover `PY` and `CMD` tags in the generated documentation, causing the output to be incorrect.

**Example Buggy Code:**
```python
def alter_interface_docs_for_cmdline(docs):
    # Original substitution regex was greedy and failed with nested brackets
    docs = re.sub(r'\[CMD:\s[^\[\]]*\sCMD\]', '', docs, flags=re.MULTILINE)
    docs = re.sub(r'\[PY:\s([^\[\]]*)\sPY\]', lambda match: match.group(1), docs, flags=re.MULTILINE)
    return docs

# Example documentation string with nested brackets
demo_doc = """
Some introductory text.

[PY: multiline python-only with [ brackets [] ] PY][CMD: multiline cli-only with [ brackets [] ] CMD]. End of demo.
"""
```

**Expected Behavior:**
The substitution process should correctly remove the `[PY: ... PY]` and `[CMD: ... CMD]` tags, even when nested brackets or newlines are present, leaving only the inner content. For example, the above `demo_doc` should be transformed to:

```
Some introductory text.

multiline python-only with [ brackets [] ] multiline cli-only with [ brackets [] ]. End of demo.
```

**Actual Behavior:**
The markup tags are not fully removed, leaving `PY` and `CMD` in the output. As a result, the transformed documentation appears as:

```
Some introductory text.

multiline python-only with [ brackets [] ] PY][CMD: multiline cli-only with [ brackets [] ] CMD]. End of demo.
```

This leads to improperly formatted documentation where the `PY` and `CMD` tags remain visible.
[/ISSUE]
{
    "test_alter_interface_docs_for_cmdline": "PASSED",
    "test_name_generation": "PASSED",
    "test_dedent": "PASSED",
    "test_alter_interface_docs_for_api": "PASSED"
}
diff --git a/datalad/cli/interface.py b/datalad/cli/interface.py
index 6a5d60212..bee96588a 100644
--- a/datalad/cli/interface.py
+++ b/datalad/cli/interface.py
@@ -60,15 +60,15 @@ def alter_interface_docs_for_cmdline(docs):
         flags=re.MULTILINE | re.DOTALL)
     # clean cmdline in-line bits
     docs = re.sub(
-        r'\[PY:\s[^\[\]]*\sPY\]',
+        r'\[PY:\s.*?\sPY\]',
         '',
         docs,
         flags=re.MULTILINE | re.DOTALL)
     docs = re.sub(
-        r'\[CMD:\s([^\[\]]*)\sCMD\]',
+        r'\[CMD:\s(.*?)\sCMD\]',
         lambda match: match.group(1),
         docs,
-        flags=re.MULTILINE)
+        flags=re.MULTILINE | re.DOTALL)
     docs = re.sub(
         r'\|\| CMDLINE \>\>(.*?)\<\< CMDLINE \|\|',
         lambda match: match.group(1),
diff --git a/datalad/interface/base.py b/datalad/interface/base.py
index 6dc512586..4bdbdd17e 100644
--- a/datalad/interface/base.py
+++ b/datalad/interface/base.py
@@ -184,15 +184,15 @@ def alter_interface_docs_for_api(docs):
         flags=re.MULTILINE | re.DOTALL)
     # clean cmdline in-line bits
     docs = re.sub(
-        r'\[CMD:\s[^\[\]]*\sCMD\]',
+        r'\[CMD:\s.*?\sCMD\]',
         '',
         docs,
         flags=re.MULTILINE | re.DOTALL)
     docs = re.sub(
-        r'\[PY:\s([^\[\]]*)\sPY\]',
+        r'\[PY:\s(.*?)\sPY\]',
         lambda match: match.group(1),
         docs,
-        flags=re.MULTILINE)
+        flags=re.MULTILINE | re.DOTALL)
     # select only the python alternative from argument specifications
     docs = re.sub(
         r'``([a-zA-Z0-9_,.]+)\|\|([a-zA-Z0-9-,.]+)``',

{"datalad/cli/interface.py": "\"\"\"Utilities and definitions for DataLad command interfaces\"\"\"\n\n# TODO this should be a dochelper\nfrom datalad.interface.base import dedent_docstring\n\n# Some known extensions and their commands to suggest whenever lookup fails\n_known_extension_commands = {\n    'datalad-container': (\n        'containers-list', 'containers-remove', 'containers-add',\n        'containers-run'),\n    'datalad-crawler': ('crawl', 'crawl-init'),\n    'datalad-deprecated': ('ls',),\n    'datalad-neuroimaging': ('bids2scidata',)\n}\n\n_deprecated_commands = {\n    'add': \"save\",\n    'uninstall': 'drop',\n}\n\n\ndef get_cmd_ex(interface):\n    \"\"\"Return the examples for the command defined by 'interface'.\n\n    Parameters\n    ----------\n    interface : subclass of Interface\n    \"\"\"\n    from datalad.interface.base import build_example\n    intf_ex = \"\\n\\n*Examples*\\n\\n\"\n    for example in interface._examples_:\n        intf_ex += build_example(example, api='cmdline')\n    return intf_ex\n\n\ndef get_cmdline_command_name(intfspec):\n    \"\"\"Given an interface specification return a cmdline command name\"\"\"\n    if len(intfspec) > 2:\n        name = intfspec[2]\n    else:\n        name = intfspec[0].split('.')[-1].replace('_', '-')\n    return name\n\n\ndef alter_interface_docs_for_cmdline(docs):\n    \"\"\"Apply modifications to interface docstrings for cmdline doc use.\"\"\"\n    # central place to alter the impression of docstrings,\n    # like removing Python API specific sections, and argument markup\n    if not docs:\n        return docs\n    import re\n    import textwrap\n\n    docs = dedent_docstring(docs)\n    # clean cmdline sections\n    docs = re.sub(\n        r'\\|\\| PYTHON \\>\\>.*?\\<\\< PYTHON \\|\\|',\n        '',\n        docs,\n        flags=re.MULTILINE | re.DOTALL)\n    # clean cmdline in-line bits\n    docs = re.sub(\n        r'\\[PY:\\s[^\\[\\]]*\\sPY\\]',\n        '',\n        docs,\n        flags=re.MULTILINE | re.DOTALL)\n    docs = re.sub(\n        r'\\[CMD:\\s([^\\[\\]]*)\\sCMD\\]',\n        lambda match: match.group(1),\n        docs,\n        flags=re.MULTILINE)\n    docs = re.sub(\n        r'\\|\\| CMDLINE \\>\\>(.*?)\\<\\< CMDLINE \\|\\|',\n        lambda match: match.group(1),\n        docs,\n        flags=re.MULTILINE | re.DOTALL)\n    # remove :role:`...` RST markup for cmdline docs\n    docs = re.sub(\n        r':\\S+:`[^`]*`[\\\\]*',\n        lambda match: ':'.join(match.group(0).split(':')[2:]).strip('`\\\\'),\n        docs,\n        flags=re.MULTILINE | re.DOTALL)\n    # make the handbook doc references more accessible\n    # the URL is a redirect configured at readthedocs\n    docs = re.sub(\n        r'(handbook:[0-9]-[0-9]*)',\n        '\\\\1 (http://handbook.datalad.org/symbols)',\n        docs)\n    # remove None constraint. In general, `None` on the cmdline means don't\n    # give option at all, but specifying `None` explicitly is practically\n    # impossible\n    docs = re.sub(\n        r',\\sor\\svalue\\smust\\sbe\\s`None`',\n        '',\n        docs,\n        flags=re.MULTILINE | re.DOTALL)\n    # capitalize variables and remove backticks to uniformize with\n    # argparse output\n    docs = re.sub(\n        r'([^`]+)`([a-zA-Z0-9_]+)`([^`]+)',\n        lambda match: f'{match.group(1)}{match.group(2).upper()}{match.group(3)}',\n        docs)\n    # select only the cmdline alternative from argument specifications\n    docs = re.sub(\n        r'``([a-zA-Z0-9_,.]+)\\|\\|([a-zA-Z0-9-,.]+)``',\n        lambda match: f'``{match.group(2)}``',\n        docs)\n    # clean up sphinx API refs\n    docs = re.sub(\n        r'\\~datalad\\.api\\.\\S*',\n        lambda match: \"`{0}`\".format(match.group(0)[13:]),\n        docs)\n    # dedicated support for version markup\n    docs = docs.replace('.. versionadded::', 'New in version')\n    docs = docs.replace('.. versionchanged::', 'Changed in version')\n    docs = docs.replace('.. deprecated::', 'Deprecated in version')\n    # Remove RST paragraph markup\n    docs = re.sub(\n        r'^.. \\S+::',\n        lambda match: match.group(0)[3:-2].upper(),\n        docs,\n        flags=re.MULTILINE)\n    docs = re.sub(\n        r'^([ ]*)\\|\\| REFLOW \\>\\>\\n(.*?)\\<\\< REFLOW \\|\\|',\n        lambda match: textwrap.fill(match.group(2), subsequent_indent=match.group(1)),\n        docs,\n        flags=re.MULTILINE | re.DOTALL)\n    return docs\n", "datalad/interface/base.py": "# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-\n# ex: set sts=4 ts=4 sw=4 et:\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n#\n#   See COPYING file distributed along with the datalad package for the\n#   copyright and license terms.\n#\n# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##\n\"\"\"High-level interface generation\n\n\"\"\"\n\n__docformat__ = 'restructuredtext'\n\nimport logging\n\nlgr = logging.getLogger('datalad.interface.base')\n\nimport os\nimport re\nimport textwrap\nimport warnings\nfrom abc import (\n    ABC,\n    abstractmethod,\n)\nfrom collections import OrderedDict\nfrom importlib import import_module\n\nimport datalad\nfrom datalad.distribution.dataset import (\n    Dataset,\n    resolve_path,\n)\nfrom datalad.interface.common_opts import eval_params\nfrom datalad.support.exceptions import CapturedException\n\ndefault_logchannels = {\n    '': 'debug',\n    'ok': 'debug',\n    'notneeded': 'debug',\n    'impossible': 'warning',\n    'error': 'error',\n}\n\n\ndef get_api_name(intfspec):\n    \"\"\"Given an interface specification return an API name for it\"\"\"\n    if len(intfspec) > 3:\n        name = intfspec[3]\n    else:\n        name = intfspec[0].split('.')[-1]\n    return name\n\n\ndef get_interface_groups(include_plugins=False):\n    \"\"\"Return a list of command groups.\n\n    Returns\n    -------\n    A list of tuples with the form (GROUP_NAME, GROUP_DESCRIPTION, COMMANDS).\n    \"\"\"\n    if include_plugins:\n        warnings.warn(\"Plugins are no longer supported.\", DeprecationWarning)\n\n    from .. import interface as _interfaces\n\n    grps = []\n    # auto detect all available interfaces and generate a function-based\n    # API from them\n    for _item in _interfaces.__dict__:\n        if not _item.startswith('_group_'):\n            continue\n        grp_name = _item[7:]\n        grp = getattr(_interfaces, _item)\n        grps.append((grp_name,) + grp)\n    return grps\n\n\ndef get_cmd_summaries(descriptions, groups, width=79):\n    \"\"\"Return summaries for the commands in `groups`.\n\n    Parameters\n    ----------\n    descriptions : dict\n        A map of group names to summaries.\n    groups : list of tuples\n        A list of groups and commands in the form described by\n        `get_interface_groups`.\n    width : int, optional\n        The maximum width of each line in the summary text.\n\n    Returns\n    -------\n    A list with a formatted entry for each command. The first command of each\n    group is preceded by an entry describing the group.\n    \"\"\"\n    cmd_summary = []\n    for grp in sorted(groups, key=lambda x: x[0]):\n        grp_descr = grp[1]\n        grp_cmds = descriptions[grp[0]]\n\n        cmd_summary.append('\\n*%s*\\n' % (grp_descr,))\n        for cd in grp_cmds:\n            cmd_summary.append('  %s\\n%s'\n                               % ((cd[0],\n                                   textwrap.fill(\n                                       cd[1].rstrip(' .'),\n                                       width - 5,\n                                       initial_indent=' ' * 6,\n                                       subsequent_indent=' ' * 6))))\n    return cmd_summary\n\n\ndef load_interface(spec):\n    \"\"\"Load and return the class for `spec`.\n\n    Parameters\n    ----------\n    spec : tuple\n        For a standard interface, the first item is the datalad source module\n        and the second object name for the interface.\n\n    Returns\n    -------\n    The interface class or, if importing the module fails, None.\n    \"\"\"\n    lgr.log(5, \"Importing module %s \", spec[0])\n    try:\n        mod = import_module(spec[0], package='datalad')\n    except Exception as e:\n        ce = CapturedException(e)\n        lgr.error(\"Internal error, cannot import interface '%s': %s\",\n                  spec[0], ce)\n        intf = None\n    else:\n        intf = getattr(mod, spec[1])\n    return intf\n\n\ndef get_cmd_doc(interface):\n    \"\"\"Return the documentation for the command defined by `interface`.\n\n    Parameters\n    ----------\n    interface : subclass of Interface\n    \"\"\"\n    intf_doc = '' if interface.__doc__ is None else interface.__doc__.strip()\n    if hasattr(interface, '_docs_'):\n        # expand docs\n        intf_doc = intf_doc.format(**interface._docs_)\n    return intf_doc\n\n\ndef dedent_docstring(text):\n    \"\"\"Remove uniform indentation from a multiline docstring\"\"\"\n    # Problem is that first line might often have no offset, so might\n    # need to be ignored from dedent call\n    if text is None:\n        return None\n    if not text.startswith(' '):\n        lines = text.split('\\n')\n        if len(lines) == 1:\n            # single line, no indentation, nothing to do\n            return text\n        text2 = '\\n'.join(lines[1:])\n        return lines[0] + \"\\n\" + textwrap.dedent(text2)\n    else:\n        return textwrap.dedent(text)\n\n\ndef alter_interface_docs_for_api(docs):\n    \"\"\"Apply modifications to interface docstrings for Python API use.\"\"\"\n    # central place to alter the impression of docstrings,\n    # like removing cmdline specific sections\n    if not docs:\n        return docs\n    docs = dedent_docstring(docs)\n    # clean cmdline sections\n    docs = re.sub(\n        r'\\|\\| CMDLINE \\>\\>.*?\\<\\< CMDLINE \\|\\|',\n        '',\n        docs,\n        flags=re.MULTILINE | re.DOTALL)\n    # clean cmdline in-line bits\n    docs = re.sub(\n        r'\\[CMD:\\s[^\\[\\]]*\\sCMD\\]',\n        '',\n        docs,\n        flags=re.MULTILINE | re.DOTALL)\n    docs = re.sub(\n        r'\\[PY:\\s([^\\[\\]]*)\\sPY\\]',\n        lambda match: match.group(1),\n        docs,\n        flags=re.MULTILINE)\n    # select only the python alternative from argument specifications\n    docs = re.sub(\n        r'``([a-zA-Z0-9_,.]+)\\|\\|([a-zA-Z0-9-,.]+)``',\n        lambda match: f'``{match.group(1)}``',\n        docs)\n    docs = re.sub(\n        r'\\|\\| PYTHON \\>\\>(.*?)\\<\\< PYTHON \\|\\|',\n        lambda match: match.group(1),\n        docs,\n        flags=re.MULTILINE | re.DOTALL)\n    if 'DATALAD_SPHINX_RUN' not in os.environ:\n        # remove :role:`...` RST markup for cmdline docs\n        docs = re.sub(\n            r':\\S+:`[^`]*`[\\\\]*',\n            lambda match: ':'.join(match.group(0).split(':')[2:]).strip('`\\\\'),\n            docs,\n            flags=re.MULTILINE | re.DOTALL)\n        # make the handbook doc references more accessible\n        # the URL is a redirect configured at readthedocs\n        docs = re.sub(\n            r'(handbook:[0-9]-[0-9]*)',\n            '\\\\1 (http://handbook.datalad.org/symbols)',\n            docs)\n    docs = re.sub(\n        r'^([ ]*)\\|\\| REFLOW \\>\\>\\n(.*?)\\<\\< REFLOW \\|\\|',\n        lambda match: textwrap.fill(match.group(2), subsequent_indent=match.group(1)),\n        docs,\n        flags=re.MULTILINE | re.DOTALL)\n    return docs\n\n\ndef is_api_arg(arg):\n    \"\"\"Return True if argument is our API argument or self or used for internal\n    purposes\n    \"\"\"\n    return arg != 'self' and not arg.startswith('_')\n\n\ndef update_docstring_with_parameters(func, params, prefix=None, suffix=None,\n                                     add_args=None):\n    \"\"\"Generate a useful docstring from a parameter spec\n\n    Amends any existing docstring of a callable with a textual\n    description of its parameters. The Parameter spec needs to match\n    the number and names of the callables arguments.\n    \"\"\"\n    from datalad.utils import getargspec\n\n    # get the signature\n    args, varargs, varkw, defaults = getargspec(func, include_kwonlyargs=True)\n    defaults = defaults or tuple()\n    if add_args:\n        add_argnames = sorted(add_args.keys())\n        args.extend(add_argnames)\n        defaults = defaults + tuple(add_args[k] for k in add_argnames)\n    ndefaults = len(defaults)\n    # start documentation with what the callable brings with it\n    doc = prefix if prefix else u''\n    if len(args) > 1:\n        if len(doc):\n            if not doc.endswith('\\n'):\n                doc += '\\n'\n            doc += '\\n'\n        doc += \"Parameters\\n----------\\n\"\n        for i, arg in enumerate(args):\n            if not is_api_arg(arg):\n                continue\n            # we need a parameter spec for each argument\n            if not arg in params:\n                raise ValueError(\"function has argument '%s' not described as a parameter\" % arg)\n            param = params[arg]\n            # validate the default -- to make sure that the parameter description is\n            # somewhat OK\n            defaults_idx = ndefaults - len(args) + i\n            if defaults_idx >= 0:\n                if param.constraints is not None:\n                    param.constraints(defaults[defaults_idx])\n            orig_docs = param._doc\n            param._doc = alter_interface_docs_for_api(param._doc)\n            doc += param.get_autodoc(\n                arg,\n                default=defaults[defaults_idx] if defaults_idx >= 0 else None,\n                has_default=defaults_idx >= 0)\n            param._doc = orig_docs\n            doc += '\\n'\n    doc += suffix if suffix else u\"\"\n    # assign the amended docs\n    func.__doc__ = doc\n    return func\n\n\n# TODO should export code_field and indicator, rather than have modes\n# TODO this should be a doc helper\ndef build_example(example, api='python'):\n    \"\"\"Build a code example.\n\n    Take a dict from a classes _example_ specification (list of dicts) and\n    build a string with an api or cmd example (for use in cmd help or\n    docstring).\n\n    Parameters\n    ----------\n    api : {'python', 'cmdline'}\n        If 'python', build Python example for docstring. If 'cmdline', build\n        cmd example.\n\n    Returns\n    -------\n    ex : str\n        Concatenated examples for the given class.\n    \"\"\"\n    if api == 'python' :\n        code_field='code_py'\n        indicator='>'\n    elif api == 'cmdline':\n        code_field='code_cmd'\n        indicator='%'\n    else:\n        raise ValueError(\"unknown API selection: {}\".format(api))\n    if code_field not in example:\n        # only show an example if it exist for the API\n        return ''\n    description = textwrap.fill(example.get('text'))\n    # this indent the code snippet to get it properly rendered as code\n    # we are not using textwrap.fill(), because it would not acknowledge\n    # any meaningful structure/formatting of code snippets. Instead, we\n    # maintain line content as is.\n    code = dedent_docstring(example.get(code_field))\n    needs_indicator = not code.startswith(indicator)\n    code = textwrap.indent(code, ' ' * (5 if needs_indicator else 3)).lstrip()\n\n    ex = \"\"\"{}::\\n\\n   {}{}\\n\\n\"\"\".format(\n        description,\n        # disable automatic prefixing, if the example already has one\n        # this enables providing more complex examples without having\n        # to infer its inner structure\n        '{} '.format(indicator)\n        if needs_indicator\n        # maintain spacing to avoid undesired relative indentation\n        else '',\n        code)\n\n    return ex\n\n\ndef update_docstring_with_examples(cls_doc, ex):\n    \"\"\"Update a commands docstring with examples.\n\n    Take _examples_ of a command, build the Python examples, and append\n    them to the docstring.\n\n    Parameters\n    ----------\n    cls_doc: str\n      docstring\n    ex: list\n      list of dicts with examples\n    \"\"\"\n    from textwrap import indent\n    if len(cls_doc):\n        cls_doc += \"\\n\"\n    cls_doc += \"    Examples\\n    --------\\n\"\n    # loop though provided examples\n    for example in ex:\n        cls_doc += indent(build_example(example, api='python'), ' '*4)\n\n    return cls_doc\n\n\ndef build_doc(cls, **kwargs):\n    \"\"\"Decorator to build docstrings for datalad commands\n\n    It's intended to decorate the class, the __call__-method of which is the\n    actual command. It expects that __call__-method to be decorated by\n    eval_results.\n\n    Note that values for any `eval_params` keys in `cls._params_` are\n    ignored.  This means one class may extend another's `_params_`\n    without worrying about filtering out `eval_params`.\n\n    Parameters\n    ----------\n    cls: Interface\n      DataLad command implementation\n    \"\"\"\n    if datalad.in_librarymode():\n        lgr.debug(\"Not assembling DataLad API docs in libary-mode\")\n        return cls\n\n    # Note, that this is a class decorator, which is executed only once when the\n    # class is imported. It builds the docstring for the class' __call__ method\n    # and returns the original class.\n    #\n    # This is because a decorator for the actual function would not be able to\n    # behave like this. To build the docstring we need to access the attribute\n    # _params of the class. From within a function decorator we cannot do this\n    # during import time, since the class is being built in this very moment and\n    # is not yet available in the module. And if we do it from within the part\n    # of a function decorator, that is executed when the function is called, we\n    # would need to actually call the command once in order to build this\n    # docstring.\n\n    lgr.debug(\"Building doc for %s\", cls)\n\n    cls_doc = cls.__doc__\n    if hasattr(cls, '_docs_'):\n        # expand docs\n        cls_doc = cls_doc.format(**cls._docs_)\n    # get examples\n    ex = getattr(cls, '_examples_', [])\n    if ex:\n        cls_doc = update_docstring_with_examples(cls_doc, ex)\n\n    call_doc = None\n    # suffix for update_docstring_with_parameters:\n    if cls.__call__.__doc__:\n        call_doc = cls.__call__.__doc__\n\n    # build standard doc and insert eval_doc\n    spec = getattr(cls, '_params_', dict())\n\n\n    # update class attributes that may override defaults\n    if not _has_eval_results_call(cls):\n        add_args = None\n    else:\n        # defaults for all common parameters are guaranteed to be available\n        # from the class\n        add_args = {k: getattr(cls, k) for k in eval_params}\n\n    # ATTN: An important consequence of this update() call is that it\n    # fulfills the docstring's promise of overriding any existing\n    # values for eval_params keys in _params_.\n    #\n\n    # get docs for eval_results parameters:\n    spec.update(eval_params)\n\n    update_docstring_with_parameters(\n        cls.__call__, spec,\n        prefix=alter_interface_docs_for_api(cls_doc),\n        suffix=alter_interface_docs_for_api(call_doc),\n        add_args=add_args\n    )\n\n    if hasattr(cls.__call__, '_dataset_method'):\n        cls.__call__._dataset_method.__doc__ = cls.__call__.__doc__\n\n    # return original\n    return cls\n\n\nclass Interface(ABC):\n    '''Abstract base class for DataLad command implementations\n\n    Any DataLad command implementation must be derived from this class. The\n    code snippet below shows a complete sketch of a Python class with such an\n    implementation.\n\n    Importantly, no instances of command classes will created. Instead the main\n    entry point is a static ``__call__()`` method, which must be implemented\n    for any command. It is incorporated as a function in :mod:`datalad.api`, by\n    default under the name of the file the implementation resides (e.g.,\n    ``command`` for a ``command.py`` file).  Therefore the file should have a\n    name that is a syntax-compliant function name. The default naming rule can\n    be overwritten with an explicit alternative name (see\n    :func:`datalad.interface.base.get_api_name`).\n\n    For commands implementing functionality that is operating on DataLad\n    datasets, a command can be also be bound to the\n    :class:`~datalad.distribution.dataset.Dataset` class as a method using\n    the ``@datasetmethod`` decorator, under the specified name.\n\n    Any ``__call__()`` implementation should be decorated with\n    :func:`datalad.interface.utils.eval_results`. This adds support for\n    standard result processing, and a range of common command parameters that\n    do not need to be manually added to the signature of ``__call__()``. Any\n    implementation decorated in this way should be implemented as a generator,\n    and ``yield`` :ref:`result records <chap_design_result_records>`.\n\n    Any argument or keyword argument that appears in the signature of\n    ``__call__()`` must have a matching item in :attr:`Interface._params_`.\n    The dictionary maps argument names to\n    :class:`datalad.support.param.Parameter` specifications. The specification\n    contain CLI argument declarations, value constraint and data type\n    conversation specifications, documentation, and optional\n    ``argparse``-specific arguments for CLI parser construction.\n\n    The class decorator :func:`datalad.interface.base.build_doc` inspects an\n    :class:`Interface` implementation, and builds a standard docstring from\n    various sources of structured information within the class (also see\n    below). The documentation is automatically tuned differently, depending on\n    the target API (Python vs CLI).\n\n    .. code:: python\n\n        @build_doc\n        class ExampleCommand(Interface):\n            \"\"\"SHORT DESCRIPTION\n\n            LONG DESCRIPTION\n            ...\n            \"\"\"\n\n            # COMMAND PARAMETER DEFINITIONS\n            _params_ = dict(\n                example=Parameter(\n                    args=(\"--example\",),\n                    doc=\"\"\"Parameter description....\"\"\",\n                    constraints=...),\n                ...\n                )\n            )\n\n            # RESULT PARAMETER OVERRIDES\n            return_type= 'list'\n            ...\n\n            # USAGE EXAMPLES\n            _examples_ = [\n                dict(text=\"Example description...\",\n                     code_py=\"Example Python code...\",\n                     code_cmd=\"Example shell code ...\"),\n                ...\n            ]\n\n            @staticmethod\n            @datasetmethod(name='example_command')\n            @eval_results\n            def __call__(example=None, ...):\n                ...\n\n                yield dict(...)\n\n    The basic implementation setup described above can be customized for\n    individual commands in various way that alter the behavior and\n    presentation of a specific command. The following overview uses\n    the code comment markers in the above snippet to illustrate where\n    in the class implementation these adjustments can be made.\n\n    (SHORT/LONG) DESCRIPTION\n\n    ``Interface.short_description`` can be defined to provide an\n    explicit short description to be used in documentation and help output,\n    replacing the auto-generated extract from the first line of the full\n    description.\n\n    COMMAND PARAMETER DEFINITIONS\n\n    When a parameter specification declares ``Parameter(args=tuple(), ...)``,\n    i.e. no arguments specified, it will be ignored by the CLI. Likewise, any\n    ``Parameter`` specification for which :func:`is_api_arg` returns ``False``\n    will also be ignored by the CLI. Additionally, any such parameter will\n    not be added to the parameter description list in the Python docstring.\n\n    RESULT PARAMETER OVERRIDES\n\n    The :func:`datalad.interface.utils.eval_results` decorator automatically\n    add a range of additional arguments to a command, which are defined in\n    :py:data:`datalad.interface.common_opts.eval_params`. For any such\n    parameter an Interface implementation can define an interface-specific\n    default value, by declaring a class member with the respective parameter\n    name and the desired default as its assigned value. This feature can be\n    used to tune the default command behavior, for example, with respect to the\n    default result rendering style, or its error behavior.\n\n    In addition to the common parameters of the Python API, an additional\n    ``Interface.result_renderer_cmdline`` can be defined, in order to\n    instruct the CLI to prefer the specified alternative result renderer\n    over an ``Interface.result_renderer`` specification.\n\n    USAGE EXAMPLES\n\n    Any number of usage examples can be described in an ``_examples_`` list\n    class attribute. Such an example contains a description, and code examples\n    for Python and CLI.\n    '''\n    _params_ = {}\n\n    @abstractmethod\n    def __call__():\n        \"\"\"Must be implemented by any command\"\"\"\n\n    # https://github.com/datalad/datalad/issues/6376\n    @classmethod\n    def get_refds_path(cls, dataset):\n        \"\"\"Return a resolved reference dataset path from a `dataset` argument\n\n        .. deprecated:: 0.16\n           Use ``require_dataset()`` instead.\n        \"\"\"\n        # theoretically a dataset could come in as a relative path -> resolve\n        if dataset is None:\n            return dataset\n        refds_path = dataset.path if isinstance(dataset, Dataset) \\\n            else Dataset(dataset).path\n        if refds_path:\n            refds_path = str(resolve_path(refds_path))\n        return refds_path\n\n\n# pull all defaults from all eval_results() related parameters and assign them\n# as attributes to the class, which then becomes the one place to query for\n# default and potential overrides\nfor k, p in eval_params.items():\n    setattr(Interface,\n            # name is always given\n            k,\n            # but there may be no default (rather unlikely, though)\n            p.cmd_kwargs.get('default', None))\n\n\ndef get_allargs_as_kwargs(call, args, kwargs):\n    \"\"\"Generate a kwargs dict from a call signature and ``*args``, ``**kwargs``\n\n    Basically resolving the argnames for all positional arguments, and\n    resolving the defaults for all kwargs that are not given in a kwargs\n    dict\n    \"\"\"\n    from datalad.utils import getargspec\n    argspec = getargspec(call, include_kwonlyargs=True)\n    defaults = argspec.defaults\n    nargs = len(argspec.args)\n    defaults = defaults or []  # ensure it is a list and not None\n    assert (nargs >= len(defaults))\n    # map any args to their name\n    argmap = list(zip(argspec.args[:len(args)], args))\n    kwargs_ = OrderedDict(argmap)\n    # map defaults of kwargs to their names (update below)\n    for k, v in zip(argspec.args[-len(defaults):], defaults):\n        if k not in kwargs_:\n            kwargs_[k] = v\n    # update with provided kwarg args\n    kwargs_.update(kwargs)\n    # XXX we cannot assert the following, because our own highlevel\n    # API commands support more kwargs than what is discoverable\n    # from their signature...\n    #assert (nargs == len(kwargs_))\n    return kwargs_\n\n\n# Only needed to support command implementations before the introduction\n# of @eval_results\ndef _has_eval_results_call(cls):\n    \"\"\"Return True if cls has a __call__ decorated with @eval_results\n    \"\"\"\n    return getattr(getattr(cls, '__call__', None), '_eval_results', False)\n"}
A user will ask you to solve a task. You should first draft your thinking process (inner monologue). Then, generate the solution.

Your response format must follow the template below:
<think>
Your thoughts or/and draft, like working through an exercise on scratch paper. Be as casual and as long as you want until you are confident to generate a correct solution.
</think>
<solution>
Final solution presented to the user.
</solution>
system
We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
[ISSUE]
**Title:** Interface substitutions fail with nested brackets in PY&CMD doc markups

**Description:**
When using nested brackets or multiline content within PY and CMD markup in documentation, the substitution process does not properly remove the markup tags. This results in leftover `PY` and `CMD` tags in the generated documentation, causing the output to be incorrect.

**Example Buggy Code:**
```python
def alter_interface_docs_for_cmdline(docs):
    # Original substitution regex was greedy and failed with nested brackets
    docs = re.sub(r'\[CMD:\s[^\[\]]*\sCMD\]', '', docs, flags=re.MULTILINE)
    docs = re.sub(r'\[PY:\s([^\[\]]*)\sPY\]', lambda match: match.group(1), docs, flags=re.MULTILINE)
    return docs

# Example documentation string with nested brackets
demo_doc = """
Some introductory text.

[PY: multiline python-only with [ brackets [] ] PY][CMD: multiline cli-only with [ brackets [] ] CMD]. End of demo.
"""
```

**Expected Behavior:**
The substitution process should correctly remove the `[PY: ... PY]` and `[CMD: ... CMD]` tags, even when nested brackets or newlines are present, leaving only the inner content. For example, the above `demo_doc` should be transformed to:

```
Some introductory text.

multiline python-only with [ brackets [] ] multiline cli-only with [ brackets [] ]. End of demo.
```

**Actual Behavior:**
The markup tags are not fully removed, leaving `PY` and `CMD` in the output. As a result, the transformed documentation appears as:

```
Some introductory text.

multiline python-only with [ brackets [] ] PY][CMD: multiline cli-only with [ brackets [] ] CMD]. End of demo.
```

This leads to improperly formatted documentation where the `PY` and `CMD` tags remain visible.
[/ISSUE]
--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.

--- BEGIN FILE ---
```
### datalad/interface/base.py
# emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*-
# ex: set sts=4 ts=4 sw=4 et:
# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
#
#   See COPYING file distributed along with the datalad package for the
#   copyright and license terms.
#
# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
"""High-level interface generation

"""

__docformat__ = 'restructuredtext'

import logging

lgr = logging.getLogger('datalad.interface.base')

import os
import re
import textwrap
import warnings
from abc import (
    ABC,
    abstractmethod,
)
from collections import OrderedDict
from importlib import import_module

import datalad
from datalad.distribution.dataset import (
    Dataset,
    resolve_path,
)
from datalad.interface.common_opts import eval_params
from datalad.support.exceptions import CapturedException

default_logchannels = {
    '': 'debug',
    'ok': 'debug',
    'notneeded': 'debug',
    'impossible': 'warning',
    'error': 'error',
}


def get_api_name(intfspec):
    """Given an interface specification return an API name for it"""
    if len(intfspec) > 3:
        name = intfspec[3]
    else:
        name = intfspec[0].split('.')[-1]
    return name


def get_interface_groups(include_plugins=False):
    """Return a list of command groups.

    Returns
    -------
    A list of tuples with the form (GROUP_NAME, GROUP_DESCRIPTION, COMMANDS).
    """
    if include_plugins:
        warnings.warn("Plugins are no longer supported.", DeprecationWarning)

    from .. import interface as _interfaces

    grps = []
    # auto detect all available interfaces and generate a function-based
    # API from them
    for _item in _interfaces.__dict__:
        if not _item.startswith('_group_'):
            continue
        grp_name = _item[7:]
        grp = getattr(_interfaces, _item)
        grps.append((grp_name,) + grp)
    return grps


def get_cmd_summaries(descriptions, groups, width=79):
    """Return summaries for the commands in `groups`.

    Parameters
    ----------
    descriptions : dict
        A map of group names to summaries.
    groups : list of tuples
        A list of groups and commands in the form described by
        `get_interface_groups`.
    width : int, optional
        The maximum width of each line in the summary text.

    Returns
    -------
    A list with a formatted entry for each command. The first command of each
    group is preceded by an entry describing the group.
    """
    cmd_summary = []
    for grp in sorted(groups, key=lambda x: x[0]):
        grp_descr = grp[1]
        grp_cmds = descriptions[grp[0]]

        cmd_summary.append('\n*%s*\n' % (grp_descr,))
        for cd in grp_cmds:
            cmd_summary.append('  %s\n%s'
                               % ((cd[0],
                                   textwrap.fill(
                                       cd[1].rstrip(' .'),
                                       width - 5,
                                       initial_indent=' ' * 6,
                                       subsequent_indent=' ' * 6))))
    return cmd_summary


def load_interface(spec):
    """Load and return the class for `spec`.

    Parameters
    ----------
    spec : tuple
        For a standard interface, the first item is the datalad source module
        and the second object name for the interface.

    Returns
    -------
    The interface class or, if importing the module fails, None.
    """
    lgr.log(5, "Importing module %s ", spec[0])
    try:
        mod = import_module(spec[0], package='datalad')
    except Exception as e:
        ce = CapturedException(e)
        lgr.error("Internal error, cannot import interface '%s': %s",
                  spec[0], ce)
        intf = None
    else:
        intf = getattr(mod, spec[1])
    return intf


def get_cmd_doc(interface):
    """Return the documentation for the command defined by `interface`.

    Parameters
    ----------
    interface : subclass of Interface
    """
    intf_doc = '' if interface.__doc__ is None else interface.__doc__.strip()
    if hasattr(interface, '_docs_'):
        # expand docs
        intf_doc = intf_doc.format(**interface._docs_)
    return intf_doc


def dedent_docstring(text):
    """Remove uniform indentation from a multiline docstring"""
    # Problem is that first line might often have no offset, so might
    # need to be ignored from dedent call
    if text is None:
        return None
    if not text.startswith(' '):
        lines = text.split('\n')
        if len(lines) == 1:
            # single line, no indentation, nothing to do
            return text
        text2 = '\n'.join(lines[1:])
        return lines[0] + "\n" + textwrap.dedent(text2)
    else:
        return textwrap.dedent(text)


def alter_interface_docs_for_api(docs):
    """Apply modifications to interface docstrings for Python API use."""
    # central place to alter the impression of docstrings,
    # like removing cmdline specific sections
    if not docs:
        return docs
    docs = dedent_docstring(docs)
    # clean cmdline sections
    docs = re.sub(
        r'\|\| CMDLINE \>\>.*?\<\< CMDLINE \|\|',
        '',
        docs,
        flags=re.MULTILINE | re.DOTALL)
    # clean cmdline in-line bits
    docs = re.sub(
        r'\[CMD:\s[^\[\]]*\sCMD\]',
        '',
        docs,
        flags=re.MULTILINE | re.DOTALL)
    docs = re.sub(
        r'\[PY:\s([^\[\]]*)\sPY\]',
        lambda match: match.group(1),
        docs,
        flags=re.MULTILINE)
    # select only the python alternative from argument specifications
    docs = re.sub(
        r'``([a-zA-Z0-9_,.]+)\|\|([a-zA-Z0-9-,.]+)``',
        lambda match: f'``{match.group(1)}``',
        docs)
    docs = re.sub(
        r'\|\| PYTHON \>\>(.*?)\<\< PYTHON \|\|',
        lambda match: match.group(1),
        docs,
        flags=re.MULTILINE | re.DOTALL)
    if 'DATALAD_SPHINX_RUN' not in os.environ:
        # remove :role:`...` RST markup for cmdline docs
        docs = re.sub(
            r':\S+:`[^`]*`[\\]*',
            lambda match: ':'.join(match.group(0).split(':')[2:]).strip('`\\'),
            docs,
            flags=re.MULTILINE | re.DOTALL)
        # make the handbook doc references more accessible
        # the URL is a redirect configured at readthedocs
        docs = re.sub(
            r'(handbook:[0-9]-[0-9]*)',
            '\\1 (http://handbook.datalad.org/symbols)',
            docs)
    docs = re.sub(
        r'^([ ]*)\|\| REFLOW \>\>\n(.*?)\<\< REFLOW \|\|',
        lambda match: textwrap.fill(match.group(2), subsequent_indent=match.group(1)),
        docs,
        flags=re.MULTILINE | re.DOTALL)
    return docs


def is_api_arg(arg):
    """Return True if argument is our API argument or self or used for internal
    purposes
    """
    return arg != 'self' and not arg.startswith('_')


def update_docstring_with_parameters(func, params, prefix=None, suffix=None,
                                     add_args=None):
    """Generate a useful docstring from a parameter spec

    Amends any existing docstring of a callable with a textual
    description of its parameters. The Parameter spec needs to match
    the number and names of the callables arguments.
    """
    from datalad.utils import getargspec

    # get the signature
    args, varargs, varkw, defaults = getargspec(func, include_kwonlyargs=True)
    defaults = defaults or tuple()
    if add_args:
        add_argnames = sorted(add_args.keys())
        args.extend(add_argnames)
        defaults = defaults + tuple(add_args[k] for k in add_argnames)
    ndefaults = len(defaults)
    # start documentation with what the callable brings with it
    doc = prefix if prefix else u''
    if len(args) > 1:
        if len(doc):
            if not doc.endswith('\n'):
                doc += '\n'
            doc += '\n'
        doc += "Parameters\n----------\n"
        for i, arg in enumerate(args):
            if not is_api_arg(arg):
                continue
            # we need a parameter spec for each argument
            if not arg in params:
                raise ValueError("function has argument '%s' not described as a parameter" % arg)
            param = params[arg]
            # validate the default -- to make sure that the parameter description is
            # somewhat OK
            defaults_idx = ndefaults - len(args) + i
            if defaults_idx >= 0:
                if param.constraints is not None:
                    param.constraints(defaults[defaults_idx])
            orig_docs = param._doc
            param._doc = alter_interface_docs_for_api(param._doc)
            doc += param.get_autodoc(
                arg,
                default=defaults[defaults_idx] if defaults_idx >= 0 else None,
                has_default=defaults_idx >= 0)
            param._doc = orig_docs
            doc += '\n'
    doc += suffix if suffix else u""
    # assign the amended docs
    func.__doc__ = doc
    return func


# TODO should export code_field and indicator, rather than have modes
# TODO this should be a doc helper
def build_example(example, api='python'):
    """Build a code example.

    Take a dict from a classes _example_ specification (list of dicts) and
    build a string with an api or cmd example (for use in cmd help or
    docstring).

    Parameters
    ----------
    api : {'python', 'cmdline'}
        If 'python', build Python example for docstring. If 'cmdline', build
        cmd example.

    Returns
    -------
    ex : str
        Concatenated examples for the given class.
    """
    if api == 'python' :
        code_field='code_py'
        indicator='>'
    elif api == 'cmdline':
        code_field='code_cmd'
        indicator='%'
    else:
        raise ValueError("unknown API selection: {}".format(api))
    if code_field not in example:
        # only show an example if it exist for the API
        return ''
    description = textwrap.fill(example.get('text'))
    # this indent the code snippet to get it properly rendered as code
    # we are not using textwrap.fill(), because it would not acknowledge
    # any meaningful structure/formatting of code snippets. Instead, we
    # maintain line content as is.
    code = dedent_docstring(example.get(code_field))
    needs_indicator = not code.startswith(indicator)
    code = textwrap.indent(code, ' ' * (5 if needs_indicator else 3)).lstrip()

    ex = """{}::\n\n   {}{}\n\n""".format(
        description,
        # disable automatic prefixing, if the example already has one
        # this enables providing more complex examples without having
        # to infer its inner structure
        '{} '.format(indicator)
        if needs_indicator
        # maintain spacing to avoid undesired relative indentation
        else '',
        code)

    return ex


def update_docstring_with_examples(cls_doc, ex):
    """Update a commands docstring with examples.

    Take _examples_ of a command, build the Python examples, and append
    them to the docstring.

    Parameters
    ----------
    cls_doc: str
      docstring
    ex: list
      list of dicts with examples
    """
    from textwrap import indent
    if len(cls_doc):
        cls_doc += "\n"
    cls_doc += "    Examples\n    --------\n"
    # loop though provided examples
    for example in ex:
        cls_doc += indent(build_example(example, api='python'), ' '*4)

    return cls_doc


def build_doc(cls, **kwargs):
    """Decorator to build docstrings for datalad commands

    It's intended to decorate the class, the __call__-method of which is the
    actual command. It expects that __call__-method to be decorated by
    eval_results.

    Note that values for any `eval_params` keys in `cls._params_` are
    ignored.  This means one class may extend another's `_params_`
    without worrying about filtering out `eval_params`.

    Parameters
    ----------
    cls: Interface
      DataLad command implementation
    """
    if datalad.in_librarymode():
        lgr.debug("Not assembling DataLad API docs in libary-mode")
        return cls

    # Note, that this is a class decorator, which is executed only once when the
    # class is imported. It builds the docstring for the class' __call__ method
    # and returns the original class.
    #
    # This is because a decorator for the actual function would not be able to
    # behave like this. To build the docstring we need to access the attribute
    # _params of the class. From within a function decorator we cannot do this
    # during import time, since the class is being built in this very moment and
    # is not yet available in the module. And if we do it from within the part
    # of a function decorator, that is executed when the function is called, we
    # would need to actually call the command once in order to build this
    # docstring.

    lgr.debug("Building doc for %s", cls)

    cls_doc = cls.__doc__
    if hasattr(cls, '_docs_'):
        # expand docs
        cls_doc = cls_doc.format(**cls._docs_)
    # get examples
    ex = getattr(cls, '_examples_', [])
    if ex:
        cls_doc = update_docstring_with_examples(cls_doc, ex)

    call_doc = None
    # suffix for update_docstring_with_parameters:
    if cls.__call__.__doc__:
        call_doc = cls.__call__.__doc__

    # build standard doc and insert eval_doc
    spec = getattr(cls, '_params_', dict())


    # update class attributes that may override defaults
    if not _has_eval_results_call(cls):
        add_args = None
    else:
        # defaults for all common parameters are guaranteed to be available
        # from the class
        add_args = {k: getattr(cls, k) for k in eval_params}

    # ATTN: An important consequence of this update() call is that it
    # fulfills the docstring's promise of overriding any existing
    # values for eval_params keys in _params_.
    #

    # get docs for eval_results parameters:
    spec.update(eval_params)

    update_docstring_with_parameters(
        cls.__call__, spec,
        prefix=alter_interface_docs_for_api(cls_doc),
        suffix=alter_interface_docs_for_api(call_doc),
        add_args=add_args
    )

    if hasattr(cls.__call__, '_dataset_method'):
        cls.__call__._dataset_method.__doc__ = cls.__call__.__doc__

    # return original
    return cls


class Interface(ABC):
    '''Abstract base class for DataLad command implementations

    Any DataLad command implementation must be derived from this class. The
    code snippet below shows a complete sketch of a Python class with such an
    implementation.

    Importantly, no instances of command classes will created. Instead the main
    entry point is a static ``__call__()`` method, which must be implemented
    for any command. It is incorporated as a function in :mod:`datalad.api`, by
    default under the name of the file the implementation resides (e.g.,
    ``command`` for a ``command.py`` file).  Therefore the file should have a
    name that is a syntax-compliant function name. The default naming rule can
    be overwritten with an explicit alternative name (see
    :func:`datalad.interface.base.get_api_name`).

    For commands implementing functionality that is operating on DataLad
    datasets, a command can be also be bound to the
    :class:`~datalad.distribution.dataset.Dataset` class as a method using
    the ``@datasetmethod`` decorator, under the specified name.

    Any ``__call__()`` implementation should be decorated with
    :func:`datalad.interface.utils.eval_results`. This adds support for
    standard result processing, and a range of common command parameters that
    do not need to be manually added to the signature of ``__call__()``. Any
    implementation decorated in this way should be implemented as a generator,
    and ``yield`` :ref:`result records <chap_design_result_records>`.

    Any argument or keyword argument that appears in the signature of
    ``__call__()`` must have a matching item in :attr:`Interface._params_`.
    The dictionary maps argument names to
    :class:`datalad.support.param.Parameter` specifications. The specification
    contain CLI argument declarations, value constraint and data type
    conversation specifications, documentation, and optional
    ``argparse``-specific arguments for CLI parser construction.

    The class decorator :func:`datalad.interface.base.build_doc` inspects an
    :class:`Interface` implementation, and builds a standard docstring from
    various sources of structured information within the class (also see
    below). The documentation is automatically tuned differently, depending on
    the target API (Python vs CLI).

    .. code:: python

        @build_doc
        class ExampleCommand(Interface):
            """SHORT DESCRIPTION

            LONG DESCRIPTION
            ...
            """

            # COMMAND PARAMETER DEFINITIONS
            _params_ = dict(
                example=Parameter(
                    args=("--example",),
                    doc="""Parameter description....""",
                    constraints=...),
                ...
                )
            )

            # RESULT PARAMETER OVERRIDES
            return_type= 'list'
            ...

            # USAGE EXAMPLES
            _examples_ = [
                dict(text="Example description...",
                     code_py="Example Python code...",
                     code_cmd="Example shell code ..."),
                ...
            ]

            @staticmethod
            @datasetmethod(name='example_command')
            @eval_results
            def __call__(example=None, ...):
                ...

                yield dict(...)

    The basic implementation setup described above can be customized for
    individual commands in various way that alter the behavior and
    presentation of a specific command. The following overview uses
    the code comment markers in the above snippet to illustrate where
    in the class implementation these adjustments can be made.

    (SHORT/LONG) DESCRIPTION

    ``Interface.short_description`` can be defined to provide an
    explicit short description to be used in documentation and help output,
    replacing the auto-generated extract from the first line of the full
    description.

    COMMAND PARAMETER DEFINITIONS

    When a parameter specification declares ``Parameter(args=tuple(), ...)``,
    i.e. no arguments specified, it will be ignored by the CLI. Likewise, any
    ``Parameter`` specification for which :func:`is_api_arg` returns ``False``
    will also be ignored by the CLI. Additionally, any such parameter will
    not be added to the parameter description list in the Python docstring.

    RESULT PARAMETER OVERRIDES

    The :func:`datalad.interface.utils.eval_results` decorator automatically
    add a range of additional arguments to a command, which are defined in
    :py:data:`datalad.interface.common_opts.eval_params`. For any such
    parameter an Interface implementation can define an interface-specific
    default value, by declaring a class member with the respective parameter
    name and the desired default as its assigned value. This feature can be
    used to tune the default command behavior, for example, with respect to the
    default result rendering style, or its error behavior.

    In addition to the common parameters of the Python API, an additional
    ``Interface.result_renderer_cmdline`` can be defined, in order to
    instruct the CLI to prefer the specified alternative result renderer
    over an ``Interface.result_renderer`` specification.

    USAGE EXAMPLES

    Any number of usage examples can be described in an ``_examples_`` list
    class attribute. Such an example contains a description, and code examples
    for Python and CLI.
    '''
    _params_ = {}

    @abstractmethod
    def __call__():
        """Must be implemented by any command"""

    # https://github.com/datalad/datalad/issues/6376
    @classmethod
    def get_refds_path(cls, dataset):
        """Return a resolved reference dataset path from a `dataset` argument

        .. deprecated:: 0.16
           Use ``require_dataset()`` instead.
        """
        # theoretically a dataset could come in as a relative path -> resolve
        if dataset is None:
            return dataset
        refds_path = dataset.path if isinstance(dataset, Dataset) \
            else Dataset(dataset).path
        if refds_path:
            refds_path = str(resolve_path(refds_path))
        return refds_path


# pull all defaults from all eval_results() related parameters and assign them
# as attributes to the class, which then becomes the one place to query for
# default and potential overrides
for k, p in eval_params.items():
    setattr(Interface,
            # name is always given
            k,
            # but there may be no default (rather unlikely, though)
            p.cmd_kwargs.get('default', None))


def get_allargs_as_kwargs(call, args, kwargs):
    """Generate a kwargs dict from a call signature and ``*args``, ``**kwargs``

    Basically resolving the argnames for all positional arguments, and
    resolving the defaults for all kwargs that are not given in a kwargs
    dict
    """
    from datalad.utils import getargspec
    argspec = getargspec(call, include_kwonlyargs=True)
    defaults = argspec.defaults
    nargs = len(argspec.args)
    defaults = defaults or []  # ensure it is a list and not None
    assert (nargs >= len(defaults))
    # map any args to their name
    argmap = list(zip(argspec.args[:len(args)], args))
    kwargs_ = OrderedDict(argmap)
    # map defaults of kwargs to their names (update below)
    for k, v in zip(argspec.args[-len(defaults):], defaults):
        if k not in kwargs_:
            kwargs_[k] = v
    # update with provided kwarg args
    kwargs_.update(kwargs)
    # XXX we cannot assert the following, because our own highlevel
    # API commands support more kwargs than what is discoverable
    # from their signature...
    #assert (nargs == len(kwargs_))
    return kwargs_


# Only needed to support command implementations before the introduction
# of @eval_results
def _has_eval_results_call(cls):
    """Return True if cls has a __call__ decorated with @eval_results
    """
    return getattr(getattr(cls, '__call__', None), '_eval_results', False)


### datalad/cli/interface.py
"""Utilities and definitions for DataLad command interfaces"""

# TODO this should be a dochelper
from datalad.interface.base import dedent_docstring

# Some known extensions and their commands to suggest whenever lookup fails
_known_extension_commands = {
    'datalad-container': (
        'containers-list', 'containers-remove', 'containers-add',
        'containers-run'),
    'datalad-crawler': ('crawl', 'crawl-init'),
    'datalad-deprecated': ('ls',),
    'datalad-neuroimaging': ('bids2scidata',)
}

_deprecated_commands = {
    'add': "save",
    'uninstall': 'drop',
}


def get_cmd_ex(interface):
    """Return the examples for the command defined by 'interface'.

    Parameters
    ----------
    interface : subclass of Interface
    """
    from datalad.interface.base import build_example
    intf_ex = "\n\n*Examples*\n\n"
    for example in interface._examples_:
        intf_ex += build_example(example, api='cmdline')
    return intf_ex


def get_cmdline_command_name(intfspec):
    """Given an interface specification return a cmdline command name"""
    if len(intfspec) > 2:
        name = intfspec[2]
    else:
        name = intfspec[0].split('.')[-1].replace('_', '-')
    return name


def alter_interface_docs_for_cmdline(docs):
    """Apply modifications to interface docstrings for cmdline doc use."""
    # central place to alter the impression of docstrings,
    # like removing Python API specific sections, and argument markup
    if not docs:
        return docs
    import re
    import textwrap

    docs = dedent_docstring(docs)
    # clean cmdline sections
    docs = re.sub(
        r'\|\| PYTHON \>\>.*?\<\< PYTHON \|\|',
        '',
        docs,
        flags=re.MULTILINE | re.DOTALL)
    # clean cmdline in-line bits
    docs = re.sub(
        r'\[PY:\s[^\[\]]*\sPY\]',
        '',
        docs,
        flags=re.MULTILINE | re.DOTALL)
    docs = re.sub(
        r'\[CMD:\s([^\[\]]*)\sCMD\]',
        lambda match: match.group(1),
        docs,
        flags=re.MULTILINE)
    docs = re.sub(
        r'\|\| CMDLINE \>\>(.*?)\<\< CMDLINE \|\|',
        lambda match: match.group(1),
        docs,
        flags=re.MULTILINE | re.DOTALL)
    # remove :role:`...` RST markup for cmdline docs
    docs = re.sub(
        r':\S+:`[^`]*`[\\]*',
        lambda match: ':'.join(match.group(0).split(':')[2:]).strip('`\\'),
        docs,
        flags=re.MULTILINE | re.DOTALL)
    # make the handbook doc references more accessible
    # the URL is a redirect configured at readthedocs
    docs = re.sub(
        r'(handbook:[0-9]-[0-9]*)',
        '\\1 (http://handbook.datalad.org/symbols)',
        docs)
    # remove None constraint. In general, `None` on the cmdline means don't
    # give option at all, but specifying `None` explicitly is practically
    # impossible
    docs = re.sub(
        r',\sor\svalue\smust\sbe\s`None`',
        '',
        docs,
        flags=re.MULTILINE | re.DOTALL)
    # capitalize variables and remove backticks to uniformize with
    # argparse output
    docs = re.sub(
        r'([^`]+)`([a-zA-Z0-9_]+)`([^`]+)',
        lambda match: f'{match.group(1)}{match.group(2).upper()}{match.group(3)}',
        docs)
    # select only the cmdline alternative from argument specifications
    docs = re.sub(
        r'``([a-zA-Z0-9_,.]+)\|\|([a-zA-Z0-9-,.]+)``',
        lambda match: f'``{match.group(2)}``',
        docs)
    # clean up sphinx API refs
    docs = re.sub(
        r'\~datalad\.api\.\S*',
        lambda match: "`{0}`".format(match.group(0)[13:]),
        docs)
    # dedicated support for version markup
    docs = docs.replace('.. versionadded::', 'New in version')
    docs = docs.replace('.. versionchanged::', 'Changed in version')
    docs = docs.replace('.. deprecated::', 'Deprecated in version')
    # Remove RST paragraph markup
    docs = re.sub(
        r'^.. \S+::',
        lambda match: match.group(0)[3:-2].upper(),
        docs,
        flags=re.MULTILINE)
    docs = re.sub(
        r'^([ ]*)\|\| REFLOW \>\>\n(.*?)\<\< REFLOW \|\|',
        lambda match: textwrap.fill(match.group(2), subsequent_indent=match.group(1)),
        docs,
        flags=re.MULTILINE | re.DOTALL)
    return docs

```
--- END FILE ---

Please first localize the bug based on the issue statement, and then generate *SEARCH/REPLACE* edits to fix the issue.

Every *SEARCH/REPLACE* edit must use this format:
1. The file path
2. The start of search block: <<<<<<< SEARCH
3. A contiguous chunk of lines to search for in the existing source code
4. The dividing line: =======
5. The lines to replace into the source code
6. The end of the replace block: >>>>>>> REPLACE

Here is an example:

```python
### mathweb/flask/app.py
<<<<<<< SEARCH
from flask import Flask
=======
import math
from flask import Flask
>>>>>>> REPLACE
```
</solution>

Please note that the *SEARCH/REPLACE* edit REQUIRES PROPER INDENTATION. If you would like to add the line '        print(x)', you must fully write that out, with all those spaces before the code!
Wrap each *SEARCH/REPLACE* edit in a code block as shown in the example above. If you have multiple *SEARCH/REPLACE* edits, use a separate code block for each one.
user