{
    "file_diffs": [
        {
            "old_file_content": "#!/usr/bin/env python\n#\n# Copyright 2009 Facebook\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"); you may\n# not use this file except in compliance with the License. You may obtain\n# a copy of the License at\n#\n#     http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT\n# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the\n# License for the specific language governing permissions and limitations\n# under the License.\n\n\"\"\"Escaping/unescaping methods for HTML, JSON, URLs, and others.\n\nAlso includes a few other miscellaneous string manipulation functions that\nhave crept in over time.\n\"\"\"\n\nfrom __future__ import absolute_import, division, print_function, with_statement\n\nimport re\nimport sys\n\nfrom tornado.util import unicode_type, basestring_type, u\n\ntry:\n    from urllib.parse import parse_qs as _parse_qs  # py3\nexcept ImportError:\n    from urlparse import parse_qs as _parse_qs  # Python 2.6+\n\ntry:\n    import htmlentitydefs  # py2\nexcept ImportError:\n    import html.entities as htmlentitydefs  # py3\n\ntry:\n    import urllib.parse as urllib_parse  # py3\nexcept ImportError:\n    import urllib as urllib_parse  # py2\n\nimport json\n\ntry:\n    unichr\nexcept NameError:\n    unichr = chr\n\n_XHTML_ESCAPE_RE = re.compile('[&<>\"\\']')\n_XHTML_ESCAPE_DICT = {'&': '&amp;', '<': '&lt;', '>': '&gt;', '\"': '&quot;',\n                      '\\'': '&#39;'}\n\n\ndef xhtml_escape(value):\n    \"\"\"Escapes a string so it is valid within HTML or XML.\n\n    Escapes the characters ``<``, ``>``, ``\"``, ``'``, and ``&``.\n    When used in attribute values the escaped strings must be enclosed\n    in quotes.\n\n    .. versionchanged:: 3.2\n\n       Added the single quote to the list of escaped characters.\n    \"\"\"\n    return _XHTML_ESCAPE_RE.sub(lambda match: _XHTML_ESCAPE_DICT[match.group(0)],\n                                to_basestring(value))\n\n\ndef xhtml_unescape(value):\n    \"\"\"Un-escapes an XML-escaped string.\"\"\"\n    return re.sub(r\"&(#?)(\\w+?);\", _convert_entity, _unicode(value))\n\n\n# The fact that json_encode wraps json.dumps is an implementation detail.\n# Please see https://github.com/tornadoweb/tornado/pull/706\n# before sending a pull request that adds **kwargs to this function.\ndef json_encode(value):\n    \"\"\"JSON-encodes the given Python object.\"\"\"\n    # JSON permits but does not require forward slashes to be escaped.\n    # This is useful when json data is emitted in a <script> tag\n    # in HTML, as it prevents </script> tags from prematurely terminating\n    # the javascript.  Some json libraries do this escaping by default,\n    # although python's standard library does not, so we do it here.\n    # http://stackoverflow.com/questions/1580647/json-why-are-forward-slashes-escaped\n    return json.dumps(value).replace(\"</\", \"<\\\\/\")\n\n\ndef json_decode(value):\n    \"\"\"Returns Python objects for the given JSON string.\"\"\"\n    return json.loads(to_basestring(value))\n\n\ndef squeeze(value):\n    \"\"\"Replace all sequences of whitespace chars with a single space.\"\"\"\n    return re.sub(r\"[\\x00-\\x20]+\", \" \", value).strip()\n\n\ndef url_escape(value, plus=True):\n    \"\"\"Returns a URL-encoded version of the given value.\n\n    If ``plus`` is true (the default), spaces will be represented\n    as \"+\" instead of \"%20\".  This is appropriate for query strings\n    but not for the path component of a URL.  Note that this default\n    is the reverse of Python's urllib module.\n\n    .. versionadded:: 3.1\n        The ``plus`` argument\n    \"\"\"\n    quote = urllib_parse.quote_plus if plus else urllib_parse.quote\n    return quote(utf8(value))\n\n\n# python 3 changed things around enough that we need two separate\n# implementations of url_unescape.  We also need our own implementation\n# of parse_qs since python 3's version insists on decoding everything.\nif sys.version_info[0] < 3:\n    def url_unescape(value, encoding='utf-8', plus=True):\n        \"\"\"Decodes the given value from a URL.\n\n        The argument may be either a byte or unicode string.\n\n        If encoding is None, the result will be a byte string.  Otherwise,\n        the result is a unicode string in the specified encoding.\n\n        If ``plus`` is true (the default), plus signs will be interpreted\n        as spaces (literal plus signs must be represented as \"%2B\").  This\n        is appropriate for query strings and form-encoded values but not\n        for the path component of a URL.  Note that this default is the\n        reverse of Python's urllib module.\n\n        .. versionadded:: 3.1\n           The ``plus`` argument\n        \"\"\"\n        unquote = (urllib_parse.unquote_plus if plus else urllib_parse.unquote)\n        if encoding is None:\n            return unquote(utf8(value))\n        else:\n            return unicode_type(unquote(utf8(value)), encoding)\n\n    parse_qs_bytes = _parse_qs\nelse:\n    def url_unescape(value, encoding='utf-8', plus=True):\n        \"\"\"Decodes the given value from a URL.\n\n        The argument may be either a byte or unicode string.\n\n        If encoding is None, the result will be a byte string.  Otherwise,\n        the result is a unicode string in the specified encoding.\n\n        If ``plus`` is true (the default), plus signs will be interpreted\n        as spaces (literal plus signs must be represented as \"%2B\").  This\n        is appropriate for query strings and form-encoded values but not\n        for the path component of a URL.  Note that this default is the\n        reverse of Python's urllib module.\n\n        .. versionadded:: 3.1\n           The ``plus`` argument\n        \"\"\"\n        if encoding is None:\n            if plus:\n                # unquote_to_bytes doesn't have a _plus variant\n                value = to_basestring(value).replace('+', ' ')\n            return urllib_parse.unquote_to_bytes(value)\n        else:\n            unquote = (urllib_parse.unquote_plus if plus\n                       else urllib_parse.unquote)\n            return unquote(to_basestring(value), encoding=encoding)\n\n    def parse_qs_bytes(qs, keep_blank_values=False, strict_parsing=False):\n        \"\"\"Parses a query string like urlparse.parse_qs, but returns the\n        values as byte strings.\n\n        Keys still become type str (interpreted as latin1 in python3!)\n        because it's too painful to keep them as byte strings in\n        python3 and in practice they're nearly always ascii anyway.\n        \"\"\"\n        # This is gross, but python3 doesn't give us another way.\n        # Latin1 is the universal donor of character encodings.\n        result = _parse_qs(qs, keep_blank_values, strict_parsing,\n                           encoding='latin1', errors='strict')\n        encoded = {}\n        for k, v in result.items():\n            encoded[k] = [i.encode('latin1') for i in v]\n        return encoded\n\n\n_UTF8_TYPES = (bytes, type(None))\n\n\ndef utf8(value):\n    \"\"\"Converts a string argument to a byte string.\n\n    If the argument is already a byte string or None, it is returned unchanged.\n    Otherwise it must be a unicode string and is encoded as utf8.\n    \"\"\"\n    if isinstance(value, _UTF8_TYPES):\n        return value\n    if not isinstance(value, unicode_type):\n        raise TypeError(\n            \"Expected bytes, unicode, or None; got %r\" % type(value)\n        )\n    return value.encode(\"utf-8\")\n\n_TO_UNICODE_TYPES = (unicode_type, type(None))\n\n\ndef to_unicode(value):\n    \"\"\"Converts a string argument to a unicode string.\n\n    If the argument is already a unicode string or None, it is returned\n    unchanged.  Otherwise it must be a byte string and is decoded as utf8.\n    \"\"\"\n    if isinstance(value, _TO_UNICODE_TYPES):\n        return value\n    if not isinstance(value, bytes):\n        raise TypeError(\n            \"Expected bytes, unicode, or None; got %r\" % type(value)\n        )\n    return value.decode(\"utf-8\")\n\n# to_unicode was previously named _unicode not because it was private,\n# but to avoid conflicts with the built-in unicode() function/type\n_unicode = to_unicode\n\n# When dealing with the standard library across python 2 and 3 it is\n# sometimes useful to have a direct conversion to the native string type\nif str is unicode_type:\n    native_str = to_unicode\nelse:\n    native_str = utf8\n\n_BASESTRING_TYPES = (basestring_type, type(None))\n\n\ndef to_basestring(value):\n    \"\"\"Converts a string argument to a subclass of basestring.\n\n    In python2, byte and unicode strings are mostly interchangeable,\n    so functions that deal with a user-supplied argument in combination\n    with ascii string constants can use either and should return the type\n    the user supplied.  In python3, the two types are not interchangeable,\n    so this method is needed to convert byte strings to unicode.\n    \"\"\"\n    if isinstance(value, _BASESTRING_TYPES):\n        return value\n    if not isinstance(value, bytes):\n        raise TypeError(\n            \"Expected bytes, unicode, or None; got %r\" % type(value)\n        )\n    return value.decode(\"utf-8\")\n\n\ndef recursive_unicode(obj):\n    \"\"\"Walks a simple data structure, converting byte strings to unicode.\n\n    Supports lists, tuples, and dictionaries.\n    \"\"\"\n    if isinstance(obj, dict):\n        return dict((recursive_unicode(k), recursive_unicode(v)) for (k, v) in obj.items())\n    elif isinstance(obj, list):\n        return list(recursive_unicode(i) for i in obj)\n    elif isinstance(obj, tuple):\n        return tuple(recursive_unicode(i) for i in obj)\n    elif isinstance(obj, bytes):\n        return to_unicode(obj)\n    else:\n        return obj\n\n# I originally used the regex from\n# http://daringfireball.net/2010/07/improved_regex_for_matching_urls\n# but it gets all exponential on certain patterns (such as too many trailing\n# dots), causing the regex matcher to never return.\n# This regex should avoid those problems.\n# Use to_unicode instead of tornado.util.u - we don't want backslashes getting\n# processed as escapes.\n_URL_RE = re.compile(to_unicode(r\"\"\"\\b((?:([\\w-]+):(/{1,3})|www[.])(?:(?:(?:[^\\s&()]|&amp;|&quot;)*(?:[^!\"#$%&'()*+,.:;<=>?@\\[\\]^`{|}~\\s]))|(?:\\((?:[^\\s&()]|&amp;|&quot;)*\\)))+)\"\"\"))\n\n\ndef linkify(text, shorten=False, extra_params=\"\",\n            require_protocol=False, permitted_protocols=[\"http\", \"https\"]):\n    \"\"\"Converts plain text into HTML with links.\n\n    For example: ``linkify(\"Hello http://tornadoweb.org!\")`` would return\n    ``Hello <a href=\"http://tornadoweb.org\">http://tornadoweb.org</a>!``\n\n    Parameters:\n\n    * ``shorten``: Long urls will be shortened for display.\n\n    * ``extra_params``: Extra text to include in the link tag, or a callable\n        taking the link as an argument and returning the extra text\n        e.g. ``linkify(text, extra_params='rel=\"nofollow\" class=\"external\"')``,\n        or::\n\n            def extra_params_cb(url):\n                if url.startswith(\"http://example.com\"):\n                    return 'class=\"internal\"'\n                else:\n                    return 'class=\"external\" rel=\"nofollow\"'\n            linkify(text, extra_params=extra_params_cb)\n\n    * ``require_protocol``: Only linkify urls which include a protocol. If\n        this is False, urls such as www.facebook.com will also be linkified.\n\n    * ``permitted_protocols``: List (or set) of protocols which should be\n        linkified, e.g. ``linkify(text, permitted_protocols=[\"http\", \"ftp\",\n        \"mailto\"])``. It is very unsafe to include protocols such as\n        ``javascript``.\n    \"\"\"\n    if extra_params and not callable(extra_params):\n        extra_params = \" \" + extra_params.strip()\n\n    def make_link(m):\n        url = m.group(1)\n        proto = m.group(2)\n        if require_protocol and not proto:\n            return url  # not protocol, no linkify\n\n        if proto and proto not in permitted_protocols:\n            return url  # bad protocol, no linkify\n\n        href = m.group(1)\n        if not proto:\n            href = \"http://\" + href   # no proto specified, use http\n\n        if callable(extra_params):\n            params = \" \" + extra_params(href).strip()\n        else:\n            params = extra_params\n\n        # clip long urls. max_len is just an approximation\n        max_len = 30\n        if shorten and len(url) > max_len:\n            before_clip = url\n            if proto:\n                proto_len = len(proto) + 1 + len(m.group(3) or \"\")  # +1 for :\n            else:\n                proto_len = 0\n\n            parts = url[proto_len:].split(\"/\")\n            if len(parts) > 1:\n                # Grab the whole host part plus the first bit of the path\n                # The path is usually not that interesting once shortened\n                # (no more slug, etc), so it really just provides a little\n                # extra indication of shortening.\n                url = url[:proto_len] + parts[0] + \"/\" + \\\n                    parts[1][:8].split('?')[0].split('.')[0]\n\n            if len(url) > max_len * 1.5:  # still too long\n                url = url[:max_len]\n\n            if url != before_clip:\n                amp = url.rfind('&')\n                # avoid splitting html char entities\n                if amp > max_len - 5:\n                    url = url[:amp]\n                url += \"...\"\n\n                if len(url) >= len(before_clip):\n                    url = before_clip\n                else:\n                    # full url is visible on mouse-over (for those who don't\n                    # have a status bar, such as Safari by default)\n                    params += ' title=\"%s\"' % href\n\n        return u('<a href=\"%s\"%s>%s</a>') % (href, params, url)\n\n    # First HTML-escape so that our strings are all safe.\n    # The regex is modified to avoid character entites other than &amp; so\n    # that we won't pick up &quot;, etc.\n    text = _unicode(xhtml_escape(text))\n    return _URL_RE.sub(make_link, text)\n\n\ndef _convert_entity(m):\n    if m.group(1) == \"#\":\n        try:\n            return unichr(int(m.group(2)))\n        except ValueError:\n            return \"&#%s;\" % m.group(2)\n    try:\n        return _HTML_UNICODE_MAP[m.group(2)]\n    except KeyError:\n        return \"&%s;\" % m.group(2)\n\n\ndef _build_unicode_map():\n    unicode_map = {}\n    for name, value in htmlentitydefs.name2codepoint.items():\n        unicode_map[name] = unichr(value)\n    return unicode_map\n\n_HTML_UNICODE_MAP = _build_unicode_map()\n",
            "new_file_content": "#!/usr/bin/env python\n#\n# Copyright 2009 Facebook\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"); you may\n# not use this file except in compliance with the License. You may obtain\n# a copy of the License at\n#\n#     http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT\n# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the\n# License for the specific language governing permissions and limitations\n# under the License.\n\n\"\"\"Escaping/unescaping methods for HTML, JSON, URLs, and others.\n\nAlso includes a few other miscellaneous string manipulation functions that\nhave crept in over time.\n\"\"\"\n\nfrom __future__ import absolute_import, division, print_function, with_statement\n\nimport re\nimport sys\n\nfrom tornado.util import unicode_type, basestring_type, u\n\ntry:\n    from urllib.parse import parse_qs as _parse_qs  # py3\nexcept ImportError:\n    from urlparse import parse_qs as _parse_qs  # Python 2.6+\n\ntry:\n    import htmlentitydefs  # py2\nexcept ImportError:\n    import html.entities as htmlentitydefs  # py3\n\ntry:\n    import urllib.parse as urllib_parse  # py3\nexcept ImportError:\n    import urllib as urllib_parse  # py2\n\nimport json\n\ntry:\n    unichr\nexcept NameError:\n    unichr = chr\n\n_XHTML_ESCAPE_RE = re.compile('[&<>\"\\']')\n_XHTML_ESCAPE_DICT = {'&': '&amp;', '<': '&lt;', '>': '&gt;', '\"': '&quot;',\n                      '\\'': '&#39;'}\n\n\ndef xhtml_escape(value):\n    \"\"\"Escapes a string so it is valid within HTML or XML.\n\n    Escapes the characters ``<``, ``>``, ``\"``, ``'``, and ``&``.\n    When used in attribute values the escaped strings must be enclosed\n    in quotes.\n\n    .. versionchanged:: 3.2\n\n       Added the single quote to the list of escaped characters.\n    \"\"\"\n    return _XHTML_ESCAPE_RE.sub(lambda match: _XHTML_ESCAPE_DICT[match.group(0)],\n                                to_basestring(value))\n\n\ndef xhtml_unescape(value):\n    \"\"\"Un-escapes an XML-escaped string.\"\"\"\n    return re.sub(r\"&(#?)(\\w+?);\", _convert_entity, _unicode(value))\n\n\n# The fact that json_encode wraps json.dumps is an implementation detail.\n# Please see https://github.com/tornadoweb/tornado/pull/706\n# before sending a pull request that adds **kwargs to this function.\ndef json_encode(value):\n    \"\"\"JSON-encodes the given Python object.\"\"\"\n    # JSON permits but does not require forward slashes to be escaped.\n    # This is useful when json data is emitted in a <script> tag\n    # in HTML, as it prevents </script> tags from prematurely terminating\n    # the javascript.  Some json libraries do this escaping by default,\n    # although python's standard library does not, so we do it here.\n    # http://stackoverflow.com/questions/1580647/json-why-are-forward-slashes-escaped\n    return json.dumps(value).replace(\"</\", \"<\\\\/\")\n\n\ndef json_decode(value):\n    \"\"\"Returns Python objects for the given JSON string.\"\"\"\n    return json.loads(to_basestring(value))\n\n\ndef squeeze(value):\n    \"\"\"Replace all sequences of whitespace chars with a single space.\"\"\"\n    return re.sub(r\"[\\x00-\\x20]+\", \" \", value).strip()\n\n\ndef url_escape(value, plus=True):\n    \"\"\"Returns a URL-encoded version of the given value.\n\n    If ``plus`` is true (the default), spaces will be represented\n    as \"+\" instead of \"%20\".  This is appropriate for query strings\n    but not for the path component of a URL.  Note that this default\n    is the reverse of Python's urllib module.\n\n    .. versionadded:: 3.1\n        The ``plus`` argument\n    \"\"\"\n    quote = urllib_parse.quote_plus if plus else urllib_parse.quote\n    return quote(utf8(value))\n\n\n# python 3 changed things around enough that we need two separate\n# implementations of url_unescape.  We also need our own implementation\n# of parse_qs since python 3's version insists on decoding everything.\nif sys.version_info[0] < 3:\n    def url_unescape(value, encoding='utf-8', plus=True):\n        \"\"\"Decodes the given value from a URL.\n\n        The argument may be either a byte or unicode string.\n\n        If encoding is None, the result will be a byte string.  Otherwise,\n        the result is a unicode string in the specified encoding.\n\n        If ``plus`` is true (the default), plus signs will be interpreted\n        as spaces (literal plus signs must be represented as \"%2B\").  This\n        is appropriate for query strings and form-encoded values but not\n        for the path component of a URL.  Note that this default is the\n        reverse of Python's urllib module.\n\n        .. versionadded:: 3.1\n           The ``plus`` argument\n        \"\"\"\n        unquote = (urllib_parse.unquote_plus if plus else urllib_parse.unquote)\n        if encoding is None:\n            return unquote(utf8(value))\n        else:\n            return unicode_type(unquote(utf8(value)), encoding)\n\n    parse_qs_bytes = _parse_qs\nelse:\n    def url_unescape(value, encoding='utf-8', plus=True):\n        \"\"\"Decodes the given value from a URL.\n\n        The argument may be either a byte or unicode string.\n\n        If encoding is None, the result will be a byte string.  Otherwise,\n        the result is a unicode string in the specified encoding.\n\n        If ``plus`` is true (the default), plus signs will be interpreted\n        as spaces (literal plus signs must be represented as \"%2B\").  This\n        is appropriate for query strings and form-encoded values but not\n        for the path component of a URL.  Note that this default is the\n        reverse of Python's urllib module.\n\n        .. versionadded:: 3.1\n           The ``plus`` argument\n        \"\"\"\n        if encoding is None:\n            if plus:\n                # unquote_to_bytes doesn't have a _plus variant\n                value = to_basestring(value).replace('+', ' ')\n            return urllib_parse.unquote_to_bytes(value)\n        else:\n            unquote = (urllib_parse.unquote_plus if plus\n                       else urllib_parse.unquote)\n            return unquote(to_basestring(value), encoding=encoding)\n\n    def parse_qs_bytes(qs, keep_blank_values=False, strict_parsing=False):\n        \"\"\"Parses a query string like urlparse.parse_qs, but returns the\n        values as byte strings.\n\n        Keys still become type str (interpreted as latin1 in python3!)\n        because it's too painful to keep them as byte strings in\n        python3 and in practice they're nearly always ascii anyway.\n        \"\"\"\n        # This is gross, but python3 doesn't give us another way.\n        # Latin1 is the universal donor of character encodings.\n        result = _parse_qs(qs, keep_blank_values, strict_parsing,\n                           encoding='latin1', errors='strict')\n        encoded = {}\n        for k, v in result.items():\n            encoded[k] = [i.encode('latin1') for i in v]\n        return encoded\n\n\n_UTF8_TYPES = (bytes, type(None))\n\n\ndef utf8(value):\n    \"\"\"Converts a string argument to a byte string.\n\n    If the argument is already a byte string or None, it is returned unchanged.\n    Otherwise it must be a unicode string and is encoded as utf8.\n    \"\"\"\n    if isinstance(value, _UTF8_TYPES):\n        return value\n    if not isinstance(value, unicode_type):\n        raise TypeError(\n            \"Expected bytes, unicode, or None; got %r\" % type(value)\n        )\n    return value.encode(\"utf-8\")\n\n_TO_UNICODE_TYPES = (unicode_type, type(None))\n\n\ndef to_unicode(value):\n    \"\"\"Converts a string argument to a unicode string.\n\n    If the argument is already a unicode string or None, it is returned\n    unchanged.  Otherwise it must be a byte string and is decoded as utf8.\n    \"\"\"\n    if isinstance(value, _TO_UNICODE_TYPES):\n        return value\n    if not isinstance(value, bytes):\n        raise TypeError(\n            \"Expected bytes, unicode, or None; got %r\" % type(value)\n        )\n    return value.decode(\"utf-8\")\n\n# to_unicode was previously named _unicode not because it was private,\n# but to avoid conflicts with the built-in unicode() function/type\n_unicode = to_unicode\n\n# When dealing with the standard library across python 2 and 3 it is\n# sometimes useful to have a direct conversion to the native string type\nif str is unicode_type:\n    native_str = to_unicode\nelse:\n    native_str = utf8\n\n_BASESTRING_TYPES = (basestring_type, type(None))\n\n\ndef to_basestring(value):\n    \"\"\"Converts a string argument to a subclass of basestring.\n\n    In python2, byte and unicode strings are mostly interchangeable,\n    so functions that deal with a user-supplied argument in combination\n    with ascii string constants can use either and should return the type\n    the user supplied.  In python3, the two types are not interchangeable,\n    so this method is needed to convert byte strings to unicode.\n    \"\"\"\n    if isinstance(value, _BASESTRING_TYPES):\n        return value\n    if not isinstance(value, bytes):\n        raise TypeError(\n            \"Expected bytes, unicode, or None; got %r\" % type(value)\n        )\n    return value.decode(\"utf-8\")\n\n\ndef recursive_unicode(obj):\n    \"\"\"Walks a simple data structure, converting byte strings to unicode.\n\n    Supports lists, tuples, and dictionaries.\n    \"\"\"\n    if isinstance(obj, dict):\n        return dict((recursive_unicode(k), recursive_unicode(v)) for (k, v) in obj.items())\n    elif isinstance(obj, list):\n        return list(recursive_unicode(i) for i in obj)\n    elif isinstance(obj, tuple):\n        return tuple(recursive_unicode(i) for i in obj)\n    elif isinstance(obj, bytes):\n        return to_unicode(obj)\n    else:\n        return obj\n\n# I originally used the regex from\n# http://daringfireball.net/2010/07/improved_regex_for_matching_urls\n# but it gets all exponential on certain patterns (such as too many trailing\n# dots), causing the regex matcher to never return.\n# This regex should avoid those problems.\n# Use to_unicode instead of tornado.util.u - we don't want backslashes getting\n# processed as escapes.\n_URL_RE = re.compile(to_unicode(r\"\"\"\\b((?:([\\w-]+):(/{1,3})|www[.])(?:(?:(?:[^\\s&()]|&amp;|&quot;)*(?:[^!\"#$%&'()*+,.:;<=>?@\\[\\]^`{|}~\\s]))|(?:\\((?:[^\\s&()]|&amp;|&quot;)*\\)))+)\"\"\"))\n\n\ndef linkify(text, shorten=False, extra_params=\"\",\n            require_protocol=False, permitted_protocols=[\"http\", \"https\"]):\n    \"\"\"Converts plain text into HTML with links.\n\n    For example: ``linkify(\"Hello http://tornadoweb.org!\")`` would return\n    ``Hello <a href=\"http://tornadoweb.org\">http://tornadoweb.org</a>!``\n\n    Parameters:\n\n    * ``shorten``: Long urls will be shortened for display.\n\n    * ``extra_params``: Extra text to include in the link tag, or a callable\n        taking the link as an argument and returning the extra text\n        e.g. ``linkify(text, extra_params='rel=\"nofollow\" class=\"external\"')``,\n        or::\n\n            def extra_params_cb(url):\n                if url.startswith(\"http://example.com\"):\n                    return 'class=\"internal\"'\n                else:\n                    return 'class=\"external\" rel=\"nofollow\"'\n            linkify(text, extra_params=extra_params_cb)\n\n    * ``require_protocol``: Only linkify urls which include a protocol. If\n        this is False, urls such as www.facebook.com will also be linkified.\n\n    * ``permitted_protocols``: List (or set) of protocols which should be\n        linkified, e.g. ``linkify(text, permitted_protocols=[\"http\", \"ftp\",\n        \"mailto\"])``. It is very unsafe to include protocols such as\n        ``javascript``.\n    \"\"\"\n    if extra_params and not callable(extra_params):\n        extra_params = \" \" + extra_params.strip()\n\n    def make_link(m):\n        url = m.group(1)\n        proto = m.group(2)\n        if require_protocol and not proto:\n            return url  # not protocol, no linkify\n\n        if proto and proto not in permitted_protocols:\n            return url  # bad protocol, no linkify\n\n        href = m.group(1)\n        if not proto:\n            href = \"http://\" + href   # no proto specified, use http\n\n        if callable(extra_params):\n            params = \" \" + extra_params(href).strip()\n        else:\n            params = extra_params\n\n        # clip long urls. max_len is just an approximation\n        max_len = 30\n        if shorten and len(url) > max_len:\n            before_clip = url\n            if proto:\n                proto_len = len(proto) + 1 + len(m.group(3) or \"\")  # +1 for :\n            else:\n                proto_len = 0\n\n            parts = url[proto_len:].split(\"/\")\n            if len(parts) > 1:\n                # Grab the whole host part plus the first bit of the path\n                # The path is usually not that interesting once shortened\n                # (no more slug, etc), so it really just provides a little\n                # extra indication of shortening.\n                url = url[:proto_len] + parts[0] + \"/\" + \\\n                    parts[1][:8].split('?')[0].split('.')[0]\n\n            if len(url) > max_len * 1.5:  # still too long\n                url = url[:max_len]\n\n            if url != before_clip:\n                amp = url.rfind('&')\n                # avoid splitting html char entities\n                if amp > max_len - 5:\n                    url = url[:amp]\n                url += \"...\"\n\n                if len(url) >= len(before_clip):\n                    url = before_clip\n                else:\n                    # full url is visible on mouse-over (for those who don't\n                    # have a status bar, such as Safari by default)\n                    params += ' title=\"%s\"' % href\n\n        return u('<a href=\"%s\"%s>%s</a>') % (href, params, url)\n\n    # First HTML-escape so that our strings are all safe.\n    # The regex is modified to avoid character entites other than &amp; so\n    # that we won't pick up &quot;, etc.\n    text = _unicode(xhtml_escape(text))\n    return _URL_RE.sub(make_link, text)\n\n\ndef _convert_entity(m):\n    if m.group(1) == \"#\":\n        try:\n            if m.group(2)[:1].lower() == 'x':\n                return unichr(int(m.group(2)[1:], 16))\n            else:\n                return unichr(int(m.group(2)))\n        except ValueError:\n            return \"&#%s;\" % m.group(2)\n    try:\n        return _HTML_UNICODE_MAP[m.group(2)]\n    except KeyError:\n        return \"&%s;\" % m.group(2)\n\n\ndef _build_unicode_map():\n    unicode_map = {}\n    for name, value in htmlentitydefs.name2codepoint.items():\n        unicode_map[name] = unichr(value)\n    return unicode_map\n\n_HTML_UNICODE_MAP = _build_unicode_map()\n",
            "header": {
                "file": {
                    "path": "tornado/escape.py"
                },
                "misc_line": null
            },
            "index_line": {
                "old_commit_hash": "2852cf51",
                "new_commit_hash": "2f04b468",
                "mode": "100644"
            },
            "is_binary_file": false,
            "binary_line": null,
            "minus_file": {
                "path": "a/tornado/escape.py"
            },
            "plus_file": {
                "path": "b/tornado/escape.py"
            },
            "hunks": [
                {
                    "descriptor": {
                        "old_range": {
                            "start": 378,
                            "length": 7
                        },
                        "new_range": {
                            "start": 378,
                            "length": 10
                        },
                        "section": "def linkify(text, shorten=False, extra_params=\"\","
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "def _convert_entity(m):",
                                "type": "context"
                            },
                            {
                                "content": "    if m.group(1) == \"#\":",
                                "type": "context"
                            },
                            {
                                "content": "        try:",
                                "type": "context"
                            },
                            {
                                "content": "            return unichr(int(m.group(2)))",
                                "type": "deleted"
                            },
                            {
                                "content": "            if m.group(2)[:1].lower() == 'x':",
                                "type": "added"
                            },
                            {
                                "content": "                return unichr(int(m.group(2)[1:], 16))",
                                "type": "added"
                            },
                            {
                                "content": "            else:",
                                "type": "added"
                            },
                            {
                                "content": "                return unichr(int(m.group(2)))",
                                "type": "added"
                            },
                            {
                                "content": "        except ValueError:",
                                "type": "context"
                            },
                            {
                                "content": "            return \"&#%s;\" % m.group(2)",
                                "type": "context"
                            },
                            {
                                "content": "    try:",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [
                        {
                            "file_name": "tornado/escape.py",
                            "type": "function",
                            "name": "_convert_entity",
                            "content": "def _convert_entity(m):\n    if m.group(1) == \"#\":\n        try:\n            return unichr(int(m.group(2)))\n        except ValueError:\n            return \"&#%s;\" % m.group(2)\n    try:\n        return _HTML_UNICODE_MAP[m.group(2)]\n    except KeyError:\n        return \"&%s;\" % m.group(2)",
                            "ast_type_str": "FunctionDef",
                            "start_lineno": 378,
                            "end_lineno": 387,
                            "parent": null
                        }
                    ],
                    "added_entities": [],
                    "deleted_entities": []
                }
            ]
        },
        {
            "old_file_content": "#!/usr/bin/env python\n\n\nfrom __future__ import absolute_import, division, print_function, with_statement\nimport tornado.escape\n\nfrom tornado.escape import utf8, xhtml_escape, xhtml_unescape, url_escape, url_unescape, to_unicode, json_decode, json_encode, squeeze, recursive_unicode\nfrom tornado.util import u, unicode_type\nfrom tornado.test.util import unittest\n\nlinkify_tests = [\n    # (input, linkify_kwargs, expected_output)\n\n    (\"hello http://world.com/!\", {},\n     u('hello <a href=\"http://world.com/\">http://world.com/</a>!')),\n\n    (\"hello http://world.com/with?param=true&stuff=yes\", {},\n     u('hello <a href=\"http://world.com/with?param=true&amp;stuff=yes\">http://world.com/with?param=true&amp;stuff=yes</a>')),\n\n    # an opened paren followed by many chars killed Gruber's regex\n    (\"http://url.com/w(aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\", {},\n     u('<a href=\"http://url.com/w\">http://url.com/w</a>(aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa')),\n\n    # as did too many dots at the end\n    (\"http://url.com/withmany.......................................\", {},\n     u('<a href=\"http://url.com/withmany\">http://url.com/withmany</a>.......................................')),\n\n    (\"http://url.com/withmany((((((((((((((((((((((((((((((((((a)\", {},\n     u('<a href=\"http://url.com/withmany\">http://url.com/withmany</a>((((((((((((((((((((((((((((((((((a)')),\n\n    # some examples from http://daringfireball.net/2009/11/liberal_regex_for_matching_urls\n    # plus a fex extras (such as multiple parentheses).\n    (\"http://foo.com/blah_blah\", {},\n     u('<a href=\"http://foo.com/blah_blah\">http://foo.com/blah_blah</a>')),\n\n    (\"http://foo.com/blah_blah/\", {},\n     u('<a href=\"http://foo.com/blah_blah/\">http://foo.com/blah_blah/</a>')),\n\n    (\"(Something like http://foo.com/blah_blah)\", {},\n     u('(Something like <a href=\"http://foo.com/blah_blah\">http://foo.com/blah_blah</a>)')),\n\n    (\"http://foo.com/blah_blah_(wikipedia)\", {},\n     u('<a href=\"http://foo.com/blah_blah_(wikipedia)\">http://foo.com/blah_blah_(wikipedia)</a>')),\n\n    (\"http://foo.com/blah_(blah)_(wikipedia)_blah\", {},\n     u('<a href=\"http://foo.com/blah_(blah)_(wikipedia)_blah\">http://foo.com/blah_(blah)_(wikipedia)_blah</a>')),\n\n    (\"(Something like http://foo.com/blah_blah_(wikipedia))\", {},\n     u('(Something like <a href=\"http://foo.com/blah_blah_(wikipedia)\">http://foo.com/blah_blah_(wikipedia)</a>)')),\n\n    (\"http://foo.com/blah_blah.\", {},\n     u('<a href=\"http://foo.com/blah_blah\">http://foo.com/blah_blah</a>.')),\n\n    (\"http://foo.com/blah_blah/.\", {},\n     u('<a href=\"http://foo.com/blah_blah/\">http://foo.com/blah_blah/</a>.')),\n\n    (\"<http://foo.com/blah_blah>\", {},\n     u('&lt;<a href=\"http://foo.com/blah_blah\">http://foo.com/blah_blah</a>&gt;')),\n\n    (\"<http://foo.com/blah_blah/>\", {},\n     u('&lt;<a href=\"http://foo.com/blah_blah/\">http://foo.com/blah_blah/</a>&gt;')),\n\n    (\"http://foo.com/blah_blah,\", {},\n     u('<a href=\"http://foo.com/blah_blah\">http://foo.com/blah_blah</a>,')),\n\n    (\"http://www.example.com/wpstyle/?p=364.\", {},\n     u('<a href=\"http://www.example.com/wpstyle/?p=364\">http://www.example.com/wpstyle/?p=364</a>.')),\n\n    (\"rdar://1234\",\n     {\"permitted_protocols\": [\"http\", \"rdar\"]},\n     u('<a href=\"rdar://1234\">rdar://1234</a>')),\n\n    (\"rdar:/1234\",\n     {\"permitted_protocols\": [\"rdar\"]},\n     u('<a href=\"rdar:/1234\">rdar:/1234</a>')),\n\n    (\"http://userid:password@example.com:8080\", {},\n     u('<a href=\"http://userid:password@example.com:8080\">http://userid:password@example.com:8080</a>')),\n\n    (\"http://userid@example.com\", {},\n     u('<a href=\"http://userid@example.com\">http://userid@example.com</a>')),\n\n    (\"http://userid@example.com:8080\", {},\n     u('<a href=\"http://userid@example.com:8080\">http://userid@example.com:8080</a>')),\n\n    (\"http://userid:password@example.com\", {},\n     u('<a href=\"http://userid:password@example.com\">http://userid:password@example.com</a>')),\n\n    (\"message://%3c330e7f8409726r6a4ba78dkf1fd71420c1bf6ff@mail.gmail.com%3e\",\n     {\"permitted_protocols\": [\"http\", \"message\"]},\n     u('<a href=\"message://%3c330e7f8409726r6a4ba78dkf1fd71420c1bf6ff@mail.gmail.com%3e\">message://%3c330e7f8409726r6a4ba78dkf1fd71420c1bf6ff@mail.gmail.com%3e</a>')),\n\n    (u(\"http://\\u27a1.ws/\\u4a39\"), {},\n     u('<a href=\"http://\\u27a1.ws/\\u4a39\">http://\\u27a1.ws/\\u4a39</a>')),\n\n    (\"<tag>http://example.com</tag>\", {},\n     u('&lt;tag&gt;<a href=\"http://example.com\">http://example.com</a>&lt;/tag&gt;')),\n\n    (\"Just a www.example.com link.\", {},\n     u('Just a <a href=\"http://www.example.com\">www.example.com</a> link.')),\n\n    (\"Just a www.example.com link.\",\n     {\"require_protocol\": True},\n     u('Just a www.example.com link.')),\n\n    (\"A http://reallylong.com/link/that/exceedsthelenglimit.html\",\n     {\"require_protocol\": True, \"shorten\": True},\n     u('A <a href=\"http://reallylong.com/link/that/exceedsthelenglimit.html\" title=\"http://reallylong.com/link/that/exceedsthelenglimit.html\">http://reallylong.com/link...</a>')),\n\n    (\"A http://reallylongdomainnamethatwillbetoolong.com/hi!\",\n     {\"shorten\": True},\n     u('A <a href=\"http://reallylongdomainnamethatwillbetoolong.com/hi\" title=\"http://reallylongdomainnamethatwillbetoolong.com/hi\">http://reallylongdomainnametha...</a>!')),\n\n    (\"A file:///passwords.txt and http://web.com link\", {},\n     u('A file:///passwords.txt and <a href=\"http://web.com\">http://web.com</a> link')),\n\n    (\"A file:///passwords.txt and http://web.com link\",\n     {\"permitted_protocols\": [\"file\"]},\n     u('A <a href=\"file:///passwords.txt\">file:///passwords.txt</a> and http://web.com link')),\n\n    (\"www.external-link.com\",\n     {\"extra_params\": 'rel=\"nofollow\" class=\"external\"'},\n     u('<a href=\"http://www.external-link.com\" rel=\"nofollow\" class=\"external\">www.external-link.com</a>')),\n\n    (\"www.external-link.com and www.internal-link.com/blogs extra\",\n     {\"extra_params\": lambda href: 'class=\"internal\"' if href.startswith(\"http://www.internal-link.com\") else 'rel=\"nofollow\" class=\"external\"'},\n     u('<a href=\"http://www.external-link.com\" rel=\"nofollow\" class=\"external\">www.external-link.com</a> and <a href=\"http://www.internal-link.com/blogs\" class=\"internal\">www.internal-link.com/blogs</a> extra')),\n\n    (\"www.external-link.com\",\n     {\"extra_params\": lambda href: '    rel=\"nofollow\" class=\"external\"  '},\n     u('<a href=\"http://www.external-link.com\" rel=\"nofollow\" class=\"external\">www.external-link.com</a>')),\n]\n\n\nclass EscapeTestCase(unittest.TestCase):\n    def test_linkify(self):\n        for text, kwargs, html in linkify_tests:\n            linked = tornado.escape.linkify(text, **kwargs)\n            self.assertEqual(linked, html)\n\n    def test_xhtml_escape(self):\n        tests = [\n            (\"<foo>\", \"&lt;foo&gt;\"),\n            (u(\"<foo>\"), u(\"&lt;foo&gt;\")),\n            (b\"<foo>\", b\"&lt;foo&gt;\"),\n\n            (\"<>&\\\"'\", \"&lt;&gt;&amp;&quot;&#39;\"),\n            (\"&amp;\", \"&amp;amp;\"),\n\n            (u(\"<\\u00e9>\"), u(\"&lt;\\u00e9&gt;\")),\n            (b\"<\\xc3\\xa9>\", b\"&lt;\\xc3\\xa9&gt;\"),\n        ]\n        for unescaped, escaped in tests:\n            self.assertEqual(utf8(xhtml_escape(unescaped)), utf8(escaped))\n            self.assertEqual(utf8(unescaped), utf8(xhtml_unescape(escaped)))\n\n    def test_url_escape_unicode(self):\n        tests = [\n            # byte strings are passed through as-is\n            (u('\\u00e9').encode('utf8'), '%C3%A9'),\n            (u('\\u00e9').encode('latin1'), '%E9'),\n\n            # unicode strings become utf8\n            (u('\\u00e9'), '%C3%A9'),\n        ]\n        for unescaped, escaped in tests:\n            self.assertEqual(url_escape(unescaped), escaped)\n\n    def test_url_unescape_unicode(self):\n        tests = [\n            ('%C3%A9', u('\\u00e9'), 'utf8'),\n            ('%C3%A9', u('\\u00c3\\u00a9'), 'latin1'),\n            ('%C3%A9', utf8(u('\\u00e9')), None),\n        ]\n        for escaped, unescaped, encoding in tests:\n            # input strings to url_unescape should only contain ascii\n            # characters, but make sure the function accepts both byte\n            # and unicode strings.\n            self.assertEqual(url_unescape(to_unicode(escaped), encoding), unescaped)\n            self.assertEqual(url_unescape(utf8(escaped), encoding), unescaped)\n\n    def test_url_escape_quote_plus(self):\n        unescaped = '+ #%'\n        plus_escaped = '%2B+%23%25'\n        escaped = '%2B%20%23%25'\n        self.assertEqual(url_escape(unescaped), plus_escaped)\n        self.assertEqual(url_escape(unescaped, plus=False), escaped)\n        self.assertEqual(url_unescape(plus_escaped), unescaped)\n        self.assertEqual(url_unescape(escaped, plus=False), unescaped)\n        self.assertEqual(url_unescape(plus_escaped, encoding=None),\n                         utf8(unescaped))\n        self.assertEqual(url_unescape(escaped, encoding=None, plus=False),\n                         utf8(unescaped))\n\n    def test_escape_return_types(self):\n        # On python2 the escape methods should generally return the same\n        # type as their argument\n        self.assertEqual(type(xhtml_escape(\"foo\")), str)\n        self.assertEqual(type(xhtml_escape(u(\"foo\"))), unicode_type)\n\n    def test_json_decode(self):\n        # json_decode accepts both bytes and unicode, but strings it returns\n        # are always unicode.\n        self.assertEqual(json_decode(b'\"foo\"'), u(\"foo\"))\n        self.assertEqual(json_decode(u('\"foo\"')), u(\"foo\"))\n\n        # Non-ascii bytes are interpreted as utf8\n        self.assertEqual(json_decode(utf8(u('\"\\u00e9\"'))), u(\"\\u00e9\"))\n\n    def test_json_encode(self):\n        # json deals with strings, not bytes.  On python 2 byte strings will\n        # convert automatically if they are utf8; on python 3 byte strings\n        # are not allowed.\n        self.assertEqual(json_decode(json_encode(u(\"\\u00e9\"))), u(\"\\u00e9\"))\n        if bytes is str:\n            self.assertEqual(json_decode(json_encode(utf8(u(\"\\u00e9\")))), u(\"\\u00e9\"))\n            self.assertRaises(UnicodeDecodeError, json_encode, b\"\\xe9\")\n\n    def test_squeeze(self):\n        self.assertEqual(squeeze(u('sequences     of    whitespace   chars')), u('sequences of whitespace chars'))\n\n    def test_recursive_unicode(self):\n        tests = {\n            'dict': {b\"foo\": b\"bar\"},\n            'list': [b\"foo\", b\"bar\"],\n            'tuple': (b\"foo\", b\"bar\"),\n            'bytes': b\"foo\"\n        }\n        self.assertEqual(recursive_unicode(tests['dict']), {u(\"foo\"): u(\"bar\")})\n        self.assertEqual(recursive_unicode(tests['list']), [u(\"foo\"), u(\"bar\")])\n        self.assertEqual(recursive_unicode(tests['tuple']), (u(\"foo\"), u(\"bar\")))\n        self.assertEqual(recursive_unicode(tests['bytes']), u(\"foo\"))\n",
            "new_file_content": "#!/usr/bin/env python\n\n\nfrom __future__ import absolute_import, division, print_function, with_statement\nimport tornado.escape\n\nfrom tornado.escape import utf8, xhtml_escape, xhtml_unescape, url_escape, url_unescape, to_unicode, json_decode, json_encode, squeeze, recursive_unicode\nfrom tornado.util import u, unicode_type\nfrom tornado.test.util import unittest\n\nlinkify_tests = [\n    # (input, linkify_kwargs, expected_output)\n\n    (\"hello http://world.com/!\", {},\n     u('hello <a href=\"http://world.com/\">http://world.com/</a>!')),\n\n    (\"hello http://world.com/with?param=true&stuff=yes\", {},\n     u('hello <a href=\"http://world.com/with?param=true&amp;stuff=yes\">http://world.com/with?param=true&amp;stuff=yes</a>')),\n\n    # an opened paren followed by many chars killed Gruber's regex\n    (\"http://url.com/w(aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\", {},\n     u('<a href=\"http://url.com/w\">http://url.com/w</a>(aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa')),\n\n    # as did too many dots at the end\n    (\"http://url.com/withmany.......................................\", {},\n     u('<a href=\"http://url.com/withmany\">http://url.com/withmany</a>.......................................')),\n\n    (\"http://url.com/withmany((((((((((((((((((((((((((((((((((a)\", {},\n     u('<a href=\"http://url.com/withmany\">http://url.com/withmany</a>((((((((((((((((((((((((((((((((((a)')),\n\n    # some examples from http://daringfireball.net/2009/11/liberal_regex_for_matching_urls\n    # plus a fex extras (such as multiple parentheses).\n    (\"http://foo.com/blah_blah\", {},\n     u('<a href=\"http://foo.com/blah_blah\">http://foo.com/blah_blah</a>')),\n\n    (\"http://foo.com/blah_blah/\", {},\n     u('<a href=\"http://foo.com/blah_blah/\">http://foo.com/blah_blah/</a>')),\n\n    (\"(Something like http://foo.com/blah_blah)\", {},\n     u('(Something like <a href=\"http://foo.com/blah_blah\">http://foo.com/blah_blah</a>)')),\n\n    (\"http://foo.com/blah_blah_(wikipedia)\", {},\n     u('<a href=\"http://foo.com/blah_blah_(wikipedia)\">http://foo.com/blah_blah_(wikipedia)</a>')),\n\n    (\"http://foo.com/blah_(blah)_(wikipedia)_blah\", {},\n     u('<a href=\"http://foo.com/blah_(blah)_(wikipedia)_blah\">http://foo.com/blah_(blah)_(wikipedia)_blah</a>')),\n\n    (\"(Something like http://foo.com/blah_blah_(wikipedia))\", {},\n     u('(Something like <a href=\"http://foo.com/blah_blah_(wikipedia)\">http://foo.com/blah_blah_(wikipedia)</a>)')),\n\n    (\"http://foo.com/blah_blah.\", {},\n     u('<a href=\"http://foo.com/blah_blah\">http://foo.com/blah_blah</a>.')),\n\n    (\"http://foo.com/blah_blah/.\", {},\n     u('<a href=\"http://foo.com/blah_blah/\">http://foo.com/blah_blah/</a>.')),\n\n    (\"<http://foo.com/blah_blah>\", {},\n     u('&lt;<a href=\"http://foo.com/blah_blah\">http://foo.com/blah_blah</a>&gt;')),\n\n    (\"<http://foo.com/blah_blah/>\", {},\n     u('&lt;<a href=\"http://foo.com/blah_blah/\">http://foo.com/blah_blah/</a>&gt;')),\n\n    (\"http://foo.com/blah_blah,\", {},\n     u('<a href=\"http://foo.com/blah_blah\">http://foo.com/blah_blah</a>,')),\n\n    (\"http://www.example.com/wpstyle/?p=364.\", {},\n     u('<a href=\"http://www.example.com/wpstyle/?p=364\">http://www.example.com/wpstyle/?p=364</a>.')),\n\n    (\"rdar://1234\",\n     {\"permitted_protocols\": [\"http\", \"rdar\"]},\n     u('<a href=\"rdar://1234\">rdar://1234</a>')),\n\n    (\"rdar:/1234\",\n     {\"permitted_protocols\": [\"rdar\"]},\n     u('<a href=\"rdar:/1234\">rdar:/1234</a>')),\n\n    (\"http://userid:password@example.com:8080\", {},\n     u('<a href=\"http://userid:password@example.com:8080\">http://userid:password@example.com:8080</a>')),\n\n    (\"http://userid@example.com\", {},\n     u('<a href=\"http://userid@example.com\">http://userid@example.com</a>')),\n\n    (\"http://userid@example.com:8080\", {},\n     u('<a href=\"http://userid@example.com:8080\">http://userid@example.com:8080</a>')),\n\n    (\"http://userid:password@example.com\", {},\n     u('<a href=\"http://userid:password@example.com\">http://userid:password@example.com</a>')),\n\n    (\"message://%3c330e7f8409726r6a4ba78dkf1fd71420c1bf6ff@mail.gmail.com%3e\",\n     {\"permitted_protocols\": [\"http\", \"message\"]},\n     u('<a href=\"message://%3c330e7f8409726r6a4ba78dkf1fd71420c1bf6ff@mail.gmail.com%3e\">message://%3c330e7f8409726r6a4ba78dkf1fd71420c1bf6ff@mail.gmail.com%3e</a>')),\n\n    (u(\"http://\\u27a1.ws/\\u4a39\"), {},\n     u('<a href=\"http://\\u27a1.ws/\\u4a39\">http://\\u27a1.ws/\\u4a39</a>')),\n\n    (\"<tag>http://example.com</tag>\", {},\n     u('&lt;tag&gt;<a href=\"http://example.com\">http://example.com</a>&lt;/tag&gt;')),\n\n    (\"Just a www.example.com link.\", {},\n     u('Just a <a href=\"http://www.example.com\">www.example.com</a> link.')),\n\n    (\"Just a www.example.com link.\",\n     {\"require_protocol\": True},\n     u('Just a www.example.com link.')),\n\n    (\"A http://reallylong.com/link/that/exceedsthelenglimit.html\",\n     {\"require_protocol\": True, \"shorten\": True},\n     u('A <a href=\"http://reallylong.com/link/that/exceedsthelenglimit.html\" title=\"http://reallylong.com/link/that/exceedsthelenglimit.html\">http://reallylong.com/link...</a>')),\n\n    (\"A http://reallylongdomainnamethatwillbetoolong.com/hi!\",\n     {\"shorten\": True},\n     u('A <a href=\"http://reallylongdomainnamethatwillbetoolong.com/hi\" title=\"http://reallylongdomainnamethatwillbetoolong.com/hi\">http://reallylongdomainnametha...</a>!')),\n\n    (\"A file:///passwords.txt and http://web.com link\", {},\n     u('A file:///passwords.txt and <a href=\"http://web.com\">http://web.com</a> link')),\n\n    (\"A file:///passwords.txt and http://web.com link\",\n     {\"permitted_protocols\": [\"file\"]},\n     u('A <a href=\"file:///passwords.txt\">file:///passwords.txt</a> and http://web.com link')),\n\n    (\"www.external-link.com\",\n     {\"extra_params\": 'rel=\"nofollow\" class=\"external\"'},\n     u('<a href=\"http://www.external-link.com\" rel=\"nofollow\" class=\"external\">www.external-link.com</a>')),\n\n    (\"www.external-link.com and www.internal-link.com/blogs extra\",\n     {\"extra_params\": lambda href: 'class=\"internal\"' if href.startswith(\"http://www.internal-link.com\") else 'rel=\"nofollow\" class=\"external\"'},\n     u('<a href=\"http://www.external-link.com\" rel=\"nofollow\" class=\"external\">www.external-link.com</a> and <a href=\"http://www.internal-link.com/blogs\" class=\"internal\">www.internal-link.com/blogs</a> extra')),\n\n    (\"www.external-link.com\",\n     {\"extra_params\": lambda href: '    rel=\"nofollow\" class=\"external\"  '},\n     u('<a href=\"http://www.external-link.com\" rel=\"nofollow\" class=\"external\">www.external-link.com</a>')),\n]\n\n\nclass EscapeTestCase(unittest.TestCase):\n    def test_linkify(self):\n        for text, kwargs, html in linkify_tests:\n            linked = tornado.escape.linkify(text, **kwargs)\n            self.assertEqual(linked, html)\n\n    def test_xhtml_escape(self):\n        tests = [\n            (\"<foo>\", \"&lt;foo&gt;\"),\n            (u(\"<foo>\"), u(\"&lt;foo&gt;\")),\n            (b\"<foo>\", b\"&lt;foo&gt;\"),\n\n            (\"<>&\\\"'\", \"&lt;&gt;&amp;&quot;&#39;\"),\n            (\"&amp;\", \"&amp;amp;\"),\n\n            (u(\"<\\u00e9>\"), u(\"&lt;\\u00e9&gt;\")),\n            (b\"<\\xc3\\xa9>\", b\"&lt;\\xc3\\xa9&gt;\"),\n        ]\n        for unescaped, escaped in tests:\n            self.assertEqual(utf8(xhtml_escape(unescaped)), utf8(escaped))\n            self.assertEqual(utf8(unescaped), utf8(xhtml_unescape(escaped)))\n\n    def test_xhtml_unescape_numeric(self):\n        tests = [\n            ('foo&#32;bar', 'foo bar'),\n            ('foo&#x20;bar', 'foo bar'),\n            ('foo&#X20;bar', 'foo bar'),\n            ('foo&#xabc;bar', u('foo\\u0abcbar')),\n            ('foo&#xyz;bar', 'foo&#xyz;bar'),  # invalid encoding\n            ('foo&#;bar', 'foo&#;bar'),        # invalid encoding\n            ('foo&#x;bar', 'foo&#x;bar'),      # invalid encoding\n        ]\n        for escaped, unescaped in tests:\n            self.assertEqual(unescaped, xhtml_unescape(escaped))\n\n    def test_url_escape_unicode(self):\n        tests = [\n            # byte strings are passed through as-is\n            (u('\\u00e9').encode('utf8'), '%C3%A9'),\n            (u('\\u00e9').encode('latin1'), '%E9'),\n\n            # unicode strings become utf8\n            (u('\\u00e9'), '%C3%A9'),\n        ]\n        for unescaped, escaped in tests:\n            self.assertEqual(url_escape(unescaped), escaped)\n\n    def test_url_unescape_unicode(self):\n        tests = [\n            ('%C3%A9', u('\\u00e9'), 'utf8'),\n            ('%C3%A9', u('\\u00c3\\u00a9'), 'latin1'),\n            ('%C3%A9', utf8(u('\\u00e9')), None),\n        ]\n        for escaped, unescaped, encoding in tests:\n            # input strings to url_unescape should only contain ascii\n            # characters, but make sure the function accepts both byte\n            # and unicode strings.\n            self.assertEqual(url_unescape(to_unicode(escaped), encoding), unescaped)\n            self.assertEqual(url_unescape(utf8(escaped), encoding), unescaped)\n\n    def test_url_escape_quote_plus(self):\n        unescaped = '+ #%'\n        plus_escaped = '%2B+%23%25'\n        escaped = '%2B%20%23%25'\n        self.assertEqual(url_escape(unescaped), plus_escaped)\n        self.assertEqual(url_escape(unescaped, plus=False), escaped)\n        self.assertEqual(url_unescape(plus_escaped), unescaped)\n        self.assertEqual(url_unescape(escaped, plus=False), unescaped)\n        self.assertEqual(url_unescape(plus_escaped, encoding=None),\n                         utf8(unescaped))\n        self.assertEqual(url_unescape(escaped, encoding=None, plus=False),\n                         utf8(unescaped))\n\n    def test_escape_return_types(self):\n        # On python2 the escape methods should generally return the same\n        # type as their argument\n        self.assertEqual(type(xhtml_escape(\"foo\")), str)\n        self.assertEqual(type(xhtml_escape(u(\"foo\"))), unicode_type)\n\n    def test_json_decode(self):\n        # json_decode accepts both bytes and unicode, but strings it returns\n        # are always unicode.\n        self.assertEqual(json_decode(b'\"foo\"'), u(\"foo\"))\n        self.assertEqual(json_decode(u('\"foo\"')), u(\"foo\"))\n\n        # Non-ascii bytes are interpreted as utf8\n        self.assertEqual(json_decode(utf8(u('\"\\u00e9\"'))), u(\"\\u00e9\"))\n\n    def test_json_encode(self):\n        # json deals with strings, not bytes.  On python 2 byte strings will\n        # convert automatically if they are utf8; on python 3 byte strings\n        # are not allowed.\n        self.assertEqual(json_decode(json_encode(u(\"\\u00e9\"))), u(\"\\u00e9\"))\n        if bytes is str:\n            self.assertEqual(json_decode(json_encode(utf8(u(\"\\u00e9\")))), u(\"\\u00e9\"))\n            self.assertRaises(UnicodeDecodeError, json_encode, b\"\\xe9\")\n\n    def test_squeeze(self):\n        self.assertEqual(squeeze(u('sequences     of    whitespace   chars')), u('sequences of whitespace chars'))\n\n    def test_recursive_unicode(self):\n        tests = {\n            'dict': {b\"foo\": b\"bar\"},\n            'list': [b\"foo\", b\"bar\"],\n            'tuple': (b\"foo\", b\"bar\"),\n            'bytes': b\"foo\"\n        }\n        self.assertEqual(recursive_unicode(tests['dict']), {u(\"foo\"): u(\"bar\")})\n        self.assertEqual(recursive_unicode(tests['list']), [u(\"foo\"), u(\"bar\")])\n        self.assertEqual(recursive_unicode(tests['tuple']), (u(\"foo\"), u(\"bar\")))\n        self.assertEqual(recursive_unicode(tests['bytes']), u(\"foo\"))\n",
            "header": {
                "file": {
                    "path": "tornado/test/escape_test.py"
                },
                "misc_line": null
            },
            "index_line": {
                "old_commit_hash": "98a23463",
                "new_commit_hash": "65765b68",
                "mode": "100644"
            },
            "is_binary_file": false,
            "binary_line": null,
            "minus_file": {
                "path": "a/tornado/test/escape_test.py"
            },
            "plus_file": {
                "path": "b/tornado/test/escape_test.py"
            },
            "hunks": [
                {
                    "descriptor": {
                        "old_range": {
                            "start": 154,
                            "length": 6
                        },
                        "new_range": {
                            "start": 154,
                            "length": 19
                        },
                        "section": "class EscapeTestCase(unittest.TestCase):"
                    },
                    "line_group": {
                        "all_lines": [
                            {
                                "content": "            self.assertEqual(utf8(xhtml_escape(unescaped)), utf8(escaped))",
                                "type": "context"
                            },
                            {
                                "content": "            self.assertEqual(utf8(unescaped), utf8(xhtml_unescape(escaped)))",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            },
                            {
                                "content": "    def test_xhtml_unescape_numeric(self):",
                                "type": "added"
                            },
                            {
                                "content": "        tests = [",
                                "type": "added"
                            },
                            {
                                "content": "            ('foo&#32;bar', 'foo bar'),",
                                "type": "added"
                            },
                            {
                                "content": "            ('foo&#x20;bar', 'foo bar'),",
                                "type": "added"
                            },
                            {
                                "content": "            ('foo&#X20;bar', 'foo bar'),",
                                "type": "added"
                            },
                            {
                                "content": "            ('foo&#xabc;bar', u('foo\\u0abcbar')),",
                                "type": "added"
                            },
                            {
                                "content": "            ('foo&#xyz;bar', 'foo&#xyz;bar'),  # invalid encoding",
                                "type": "added"
                            },
                            {
                                "content": "            ('foo&#;bar', 'foo&#;bar'),        # invalid encoding",
                                "type": "added"
                            },
                            {
                                "content": "            ('foo&#x;bar', 'foo&#x;bar'),      # invalid encoding",
                                "type": "added"
                            },
                            {
                                "content": "        ]",
                                "type": "added"
                            },
                            {
                                "content": "        for escaped, unescaped in tests:",
                                "type": "added"
                            },
                            {
                                "content": "            self.assertEqual(unescaped, xhtml_unescape(escaped))",
                                "type": "added"
                            },
                            {
                                "content": "",
                                "type": "added"
                            },
                            {
                                "content": "    def test_url_escape_unicode(self):",
                                "type": "context"
                            },
                            {
                                "content": "        tests = [",
                                "type": "context"
                            },
                            {
                                "content": "            # byte strings are passed through as-is",
                                "type": "context"
                            },
                            {
                                "content": "",
                                "type": "context"
                            }
                        ]
                    },
                    "modified_entities": [
                        {
                            "file_name": "tornado/test/escape_test.py",
                            "type": "class",
                            "name": "EscapeTestCase",
                            "content": "class EscapeTestCase(unittest.TestCase):\n    def test_linkify(self):\n        for text, kwargs, html in linkify_tests:\n            linked = tornado.escape.linkify(text, **kwargs)\n            self.assertEqual(linked, html)\n\n    def test_xhtml_escape(self):\n        tests = [\n            (\"<foo>\", \"&lt;foo&gt;\"),\n            (u(\"<foo>\"), u(\"&lt;foo&gt;\")),\n            (b\"<foo>\", b\"&lt;foo&gt;\"),\n\n            (\"<>&\\\"'\", \"&lt;&gt;&amp;&quot;&#39;\"),\n            (\"&amp;\", \"&amp;amp;\"),\n\n            (u(\"<\\u00e9>\"), u(\"&lt;\\u00e9&gt;\")),\n            (b\"<\\xc3\\xa9>\", b\"&lt;\\xc3\\xa9&gt;\"),\n        ]\n        for unescaped, escaped in tests:\n            self.assertEqual(utf8(xhtml_escape(unescaped)), utf8(escaped))\n            self.assertEqual(utf8(unescaped), utf8(xhtml_unescape(escaped)))\n\n    def test_xhtml_unescape_numeric(self):\n        tests = [\n            ('foo&#32;bar', 'foo bar'),\n            ('foo&#x20;bar', 'foo bar'),\n            ('foo&#X20;bar', 'foo bar'),\n            ('foo&#xabc;bar', u('foo\\u0abcbar')),\n            ('foo&#xyz;bar', 'foo&#xyz;bar'),  # invalid encoding\n            ('foo&#;bar', 'foo&#;bar'),        # invalid encoding\n            ('foo&#x;bar', 'foo&#x;bar'),      # invalid encoding\n        ]\n        for escaped, unescaped in tests:\n            self.assertEqual(unescaped, xhtml_unescape(escaped))\n\n    def test_url_escape_unicode(self):\n        tests = [\n            # byte strings are passed through as-is\n            (u('\\u00e9').encode('utf8'), '%C3%A9'),\n            (u('\\u00e9').encode('latin1'), '%E9'),\n\n            # unicode strings become utf8\n            (u('\\u00e9'), '%C3%A9'),\n        ]\n        for unescaped, escaped in tests:\n            self.assertEqual(url_escape(unescaped), escaped)\n\n    def test_url_unescape_unicode(self):\n        tests = [\n            ('%C3%A9', u('\\u00e9'), 'utf8'),\n            ('%C3%A9', u('\\u00c3\\u00a9'), 'latin1'),\n            ('%C3%A9', utf8(u('\\u00e9')), None),\n        ]\n        for escaped, unescaped, encoding in tests:\n            # input strings to url_unescape should only contain ascii\n            # characters, but make sure the function accepts both byte\n            # and unicode strings.\n            self.assertEqual(url_unescape(to_unicode(escaped), encoding), unescaped)\n            self.assertEqual(url_unescape(utf8(escaped), encoding), unescaped)\n\n    def test_url_escape_quote_plus(self):\n        unescaped = '+ #%'\n        plus_escaped = '%2B+%23%25'\n        escaped = '%2B%20%23%25'\n        self.assertEqual(url_escape(unescaped), plus_escaped)\n        self.assertEqual(url_escape(unescaped, plus=False), escaped)\n        self.assertEqual(url_unescape(plus_escaped), unescaped)\n        self.assertEqual(url_unescape(escaped, plus=False), unescaped)\n        self.assertEqual(url_unescape(plus_escaped, encoding=None),\n                         utf8(unescaped))\n        self.assertEqual(url_unescape(escaped, encoding=None, plus=False),\n                         utf8(unescaped))\n\n    def test_escape_return_types(self):\n        # On python2 the escape methods should generally return the same\n        # type as their argument\n        self.assertEqual(type(xhtml_escape(\"foo\")), str)\n        self.assertEqual(type(xhtml_escape(u(\"foo\"))), unicode_type)\n\n    def test_json_decode(self):\n        # json_decode accepts both bytes and unicode, but strings it returns\n        # are always unicode.\n        self.assertEqual(json_decode(b'\"foo\"'), u(\"foo\"))\n        self.assertEqual(json_decode(u('\"foo\"')), u(\"foo\"))\n\n        # Non-ascii bytes are interpreted as utf8\n        self.assertEqual(json_decode(utf8(u('\"\\u00e9\"'))), u(\"\\u00e9\"))\n\n    def test_json_encode(self):\n        # json deals with strings, not bytes.  On python 2 byte strings will\n        # convert automatically if they are utf8; on python 3 byte strings\n        # are not allowed.\n        self.assertEqual(json_decode(json_encode(u(\"\\u00e9\"))), u(\"\\u00e9\"))\n        if bytes is str:\n            self.assertEqual(json_decode(json_encode(utf8(u(\"\\u00e9\")))), u(\"\\u00e9\"))\n            self.assertRaises(UnicodeDecodeError, json_encode, b\"\\xe9\")\n\n    def test_squeeze(self):\n        self.assertEqual(squeeze(u('sequences     of    whitespace   chars')), u('sequences of whitespace chars'))\n\n    def test_recursive_unicode(self):\n        tests = {\n            'dict': {b\"foo\": b\"bar\"},\n            'list': [b\"foo\", b\"bar\"],\n            'tuple': (b\"foo\", b\"bar\"),\n            'bytes': b\"foo\"\n        }\n        self.assertEqual(recursive_unicode(tests['dict']), {u(\"foo\"): u(\"bar\")})\n        self.assertEqual(recursive_unicode(tests['list']), [u(\"foo\"), u(\"bar\")])\n        self.assertEqual(recursive_unicode(tests['tuple']), (u(\"foo\"), u(\"bar\")))\n        self.assertEqual(recursive_unicode(tests['bytes']), u(\"foo\"))",
                            "ast_type_str": "ClassDef",
                            "start_lineno": 135,
                            "end_lineno": 245,
                            "parent": null
                        }
                    ],
                    "added_entities": [
                        {
                            "file_name": "tornado/test/escape_test.py",
                            "type": "method",
                            "name": "EscapeTestCase.test_xhtml_unescape_numeric",
                            "content": "class EscapeTestCase(unittest.TestCase):\n    def test_linkify(self):\n        for text, kwargs, html in linkify_tests:\n            linked = tornado.escape.linkify(text, **kwargs)\n            self.assertEqual(linked, html)\n\n    def test_xhtml_escape(self):\n        tests = [\n            (\"<foo>\", \"&lt;foo&gt;\"),\n            (u(\"<foo>\"), u(\"&lt;foo&gt;\")),\n            (b\"<foo>\", b\"&lt;foo&gt;\"),\n\n            (\"<>&\\\"'\", \"&lt;&gt;&amp;&quot;&#39;\"),\n            (\"&amp;\", \"&amp;amp;\"),\n\n            (u(\"<\\u00e9>\"), u(\"&lt;\\u00e9&gt;\")),\n            (b\"<\\xc3\\xa9>\", b\"&lt;\\xc3\\xa9&gt;\"),\n        ]\n        for unescaped, escaped in tests:\n            self.assertEqual(utf8(xhtml_escape(unescaped)), utf8(escaped))\n            self.assertEqual(utf8(unescaped), utf8(xhtml_unescape(escaped)))\n\n    def test_xhtml_unescape_numeric(self):\n        tests = [\n            ('foo&#32;bar', 'foo bar'),\n            ('foo&#x20;bar', 'foo bar'),\n            ('foo&#X20;bar', 'foo bar'),\n            ('foo&#xabc;bar', u('foo\\u0abcbar')),\n            ('foo&#xyz;bar', 'foo&#xyz;bar'),  # invalid encoding\n            ('foo&#;bar', 'foo&#;bar'),        # invalid encoding\n            ('foo&#x;bar', 'foo&#x;bar'),      # invalid encoding\n        ]\n        for escaped, unescaped in tests:\n            self.assertEqual(unescaped, xhtml_unescape(escaped))\n\n    def test_url_escape_unicode(self):\n        tests = [\n            # byte strings are passed through as-is\n            (u('\\u00e9').encode('utf8'), '%C3%A9'),\n            (u('\\u00e9').encode('latin1'), '%E9'),\n\n            # unicode strings become utf8\n            (u('\\u00e9'), '%C3%A9'),\n        ]\n        for unescaped, escaped in tests:\n            self.assertEqual(url_escape(unescaped), escaped)\n\n    def test_url_unescape_unicode(self):\n        tests = [\n            ('%C3%A9', u('\\u00e9'), 'utf8'),\n            ('%C3%A9', u('\\u00c3\\u00a9'), 'latin1'),\n            ('%C3%A9', utf8(u('\\u00e9')), None),\n        ]\n        for escaped, unescaped, encoding in tests:\n            # input strings to url_unescape should only contain ascii\n            # characters, but make sure the function accepts both byte\n            # and unicode strings.\n            self.assertEqual(url_unescape(to_unicode(escaped), encoding), unescaped)\n            self.assertEqual(url_unescape(utf8(escaped), encoding), unescaped)\n\n    def test_url_escape_quote_plus(self):\n        unescaped = '+ #%'\n        plus_escaped = '%2B+%23%25'\n        escaped = '%2B%20%23%25'\n        self.assertEqual(url_escape(unescaped), plus_escaped)\n        self.assertEqual(url_escape(unescaped, plus=False), escaped)\n        self.assertEqual(url_unescape(plus_escaped), unescaped)\n        self.assertEqual(url_unescape(escaped, plus=False), unescaped)\n        self.assertEqual(url_unescape(plus_escaped, encoding=None),\n                         utf8(unescaped))\n        self.assertEqual(url_unescape(escaped, encoding=None, plus=False),\n                         utf8(unescaped))\n\n    def test_escape_return_types(self):\n        # On python2 the escape methods should generally return the same\n        # type as their argument\n        self.assertEqual(type(xhtml_escape(\"foo\")), str)\n        self.assertEqual(type(xhtml_escape(u(\"foo\"))), unicode_type)\n\n    def test_json_decode(self):\n        # json_decode accepts both bytes and unicode, but strings it returns\n        # are always unicode.\n        self.assertEqual(json_decode(b'\"foo\"'), u(\"foo\"))\n        self.assertEqual(json_decode(u('\"foo\"')), u(\"foo\"))\n\n        # Non-ascii bytes are interpreted as utf8\n        self.assertEqual(json_decode(utf8(u('\"\\u00e9\"'))), u(\"\\u00e9\"))\n\n    def test_json_encode(self):\n        # json deals with strings, not bytes.  On python 2 byte strings will\n        # convert automatically if they are utf8; on python 3 byte strings\n        # are not allowed.\n        self.assertEqual(json_decode(json_encode(u(\"\\u00e9\"))), u(\"\\u00e9\"))\n        if bytes is str:\n            self.assertEqual(json_decode(json_encode(utf8(u(\"\\u00e9\")))), u(\"\\u00e9\"))\n            self.assertRaises(UnicodeDecodeError, json_encode, b\"\\xe9\")\n\n    def test_squeeze(self):\n        self.assertEqual(squeeze(u('sequences     of    whitespace   chars')), u('sequences of whitespace chars'))\n\n    def test_recursive_unicode(self):\n        tests = {\n            'dict': {b\"foo\": b\"bar\"},\n            'list': [b\"foo\", b\"bar\"],\n            'tuple': (b\"foo\", b\"bar\"),\n            'bytes': b\"foo\"\n        }\n        self.assertEqual(recursive_unicode(tests['dict']), {u(\"foo\"): u(\"bar\")})\n        self.assertEqual(recursive_unicode(tests['list']), [u(\"foo\"), u(\"bar\")])\n        self.assertEqual(recursive_unicode(tests['tuple']), (u(\"foo\"), u(\"bar\")))\n        self.assertEqual(recursive_unicode(tests['bytes']), u(\"foo\"))",
                            "ast_type_str": "ClassDef",
                            "start_lineno": 157,
                            "end_lineno": 168,
                            "parent": {
                                "file_name": "tornado/test/escape_test.py",
                                "type": "class",
                                "name": "EscapeTestCase",
                                "content": "class EscapeTestCase(unittest.TestCase):\n    def test_linkify(self):\n        for text, kwargs, html in linkify_tests:\n            linked = tornado.escape.linkify(text, **kwargs)\n            self.assertEqual(linked, html)\n\n    def test_xhtml_escape(self):\n        tests = [\n            (\"<foo>\", \"&lt;foo&gt;\"),\n            (u(\"<foo>\"), u(\"&lt;foo&gt;\")),\n            (b\"<foo>\", b\"&lt;foo&gt;\"),\n\n            (\"<>&\\\"'\", \"&lt;&gt;&amp;&quot;&#39;\"),\n            (\"&amp;\", \"&amp;amp;\"),\n\n            (u(\"<\\u00e9>\"), u(\"&lt;\\u00e9&gt;\")),\n            (b\"<\\xc3\\xa9>\", b\"&lt;\\xc3\\xa9&gt;\"),\n        ]\n        for unescaped, escaped in tests:\n            self.assertEqual(utf8(xhtml_escape(unescaped)), utf8(escaped))\n            self.assertEqual(utf8(unescaped), utf8(xhtml_unescape(escaped)))\n\n    def test_xhtml_unescape_numeric(self):\n        tests = [\n            ('foo&#32;bar', 'foo bar'),\n            ('foo&#x20;bar', 'foo bar'),\n            ('foo&#X20;bar', 'foo bar'),\n            ('foo&#xabc;bar', u('foo\\u0abcbar')),\n            ('foo&#xyz;bar', 'foo&#xyz;bar'),  # invalid encoding\n            ('foo&#;bar', 'foo&#;bar'),        # invalid encoding\n            ('foo&#x;bar', 'foo&#x;bar'),      # invalid encoding\n        ]\n        for escaped, unescaped in tests:\n            self.assertEqual(unescaped, xhtml_unescape(escaped))\n\n    def test_url_escape_unicode(self):\n        tests = [\n            # byte strings are passed through as-is\n            (u('\\u00e9').encode('utf8'), '%C3%A9'),\n            (u('\\u00e9').encode('latin1'), '%E9'),\n\n            # unicode strings become utf8\n            (u('\\u00e9'), '%C3%A9'),\n        ]\n        for unescaped, escaped in tests:\n            self.assertEqual(url_escape(unescaped), escaped)\n\n    def test_url_unescape_unicode(self):\n        tests = [\n            ('%C3%A9', u('\\u00e9'), 'utf8'),\n            ('%C3%A9', u('\\u00c3\\u00a9'), 'latin1'),\n            ('%C3%A9', utf8(u('\\u00e9')), None),\n        ]\n        for escaped, unescaped, encoding in tests:\n            # input strings to url_unescape should only contain ascii\n            # characters, but make sure the function accepts both byte\n            # and unicode strings.\n            self.assertEqual(url_unescape(to_unicode(escaped), encoding), unescaped)\n            self.assertEqual(url_unescape(utf8(escaped), encoding), unescaped)\n\n    def test_url_escape_quote_plus(self):\n        unescaped = '+ #%'\n        plus_escaped = '%2B+%23%25'\n        escaped = '%2B%20%23%25'\n        self.assertEqual(url_escape(unescaped), plus_escaped)\n        self.assertEqual(url_escape(unescaped, plus=False), escaped)\n        self.assertEqual(url_unescape(plus_escaped), unescaped)\n        self.assertEqual(url_unescape(escaped, plus=False), unescaped)\n        self.assertEqual(url_unescape(plus_escaped, encoding=None),\n                         utf8(unescaped))\n        self.assertEqual(url_unescape(escaped, encoding=None, plus=False),\n                         utf8(unescaped))\n\n    def test_escape_return_types(self):\n        # On python2 the escape methods should generally return the same\n        # type as their argument\n        self.assertEqual(type(xhtml_escape(\"foo\")), str)\n        self.assertEqual(type(xhtml_escape(u(\"foo\"))), unicode_type)\n\n    def test_json_decode(self):\n        # json_decode accepts both bytes and unicode, but strings it returns\n        # are always unicode.\n        self.assertEqual(json_decode(b'\"foo\"'), u(\"foo\"))\n        self.assertEqual(json_decode(u('\"foo\"')), u(\"foo\"))\n\n        # Non-ascii bytes are interpreted as utf8\n        self.assertEqual(json_decode(utf8(u('\"\\u00e9\"'))), u(\"\\u00e9\"))\n\n    def test_json_encode(self):\n        # json deals with strings, not bytes.  On python 2 byte strings will\n        # convert automatically if they are utf8; on python 3 byte strings\n        # are not allowed.\n        self.assertEqual(json_decode(json_encode(u(\"\\u00e9\"))), u(\"\\u00e9\"))\n        if bytes is str:\n            self.assertEqual(json_decode(json_encode(utf8(u(\"\\u00e9\")))), u(\"\\u00e9\"))\n            self.assertRaises(UnicodeDecodeError, json_encode, b\"\\xe9\")\n\n    def test_squeeze(self):\n        self.assertEqual(squeeze(u('sequences     of    whitespace   chars')), u('sequences of whitespace chars'))\n\n    def test_recursive_unicode(self):\n        tests = {\n            'dict': {b\"foo\": b\"bar\"},\n            'list': [b\"foo\", b\"bar\"],\n            'tuple': (b\"foo\", b\"bar\"),\n            'bytes': b\"foo\"\n        }\n        self.assertEqual(recursive_unicode(tests['dict']), {u(\"foo\"): u(\"bar\")})\n        self.assertEqual(recursive_unicode(tests['list']), [u(\"foo\"), u(\"bar\")])\n        self.assertEqual(recursive_unicode(tests['tuple']), (u(\"foo\"), u(\"bar\")))\n        self.assertEqual(recursive_unicode(tests['bytes']), u(\"foo\"))",
                                "ast_type_str": "ClassDef",
                                "start_lineno": 135,
                                "end_lineno": 245,
                                "parent": null
                            }
                        }
                    ],
                    "deleted_entities": []
                }
            ]
        }
    ],
    "old_commit_hash": "598ffdace41c4ed96eceb98cfcd27388d2f11657^",
    "new_commit_hash": "598ffdace41c4ed96eceb98cfcd27388d2f11657",
    "commit_message": "Modify xhtml_unescape to handle hex numeric refs\n\nWhile the existing code handles references like &#32; properly,\nit doesn't handle the valid case where the number is specified\nin hex. For example 'foo&#32;bar', 'foo&#x20;bar' and 'foo&#X20;bar'\nshould all decode to 'foo bar'. The changes in escape.py check\nfor the 'x' and decode appropriately. Also added unit tests for\nunescaping strings with numeric references.",
    "commit_date": "2015-04-24T12:23:48-04:00",
    "metadata": {}
}
{
    "repo_name": "tornado",
    "new_commit_hash": "598ffdace41c4ed96eceb98cfcd27388d2f11657",
    "test_file_codes": [
        "#!/usr/bin/env python\n\n\nfrom __future__ import absolute_import, division, print_function, with_statement\nimport tornado.escape\n\nfrom tornado.escape import utf8, xhtml_escape, xhtml_unescape, url_escape, url_unescape, to_unicode, json_decode, json_encode, squeeze, recursive_unicode\nfrom tornado.util import u, unicode_type\nfrom tornado.test.util import unittest\n\nlinkify_tests = [\n    # (input, linkify_kwargs, expected_output)\n\n    (\"hello http://world.com/!\", {},\n     u('hello <a href=\"http://world.com/\">http://world.com/</a>!')),\n\n    (\"hello http://world.com/with?param=true&stuff=yes\", {},\n     u('hello <a href=\"http://world.com/with?param=true&amp;stuff=yes\">http://world.com/with?param=true&amp;stuff=yes</a>')),\n\n    # an opened paren followed by many chars killed Gruber's regex\n    (\"http://url.com/w(aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\", {},\n     u('<a href=\"http://url.com/w\">http://url.com/w</a>(aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa')),\n\n    # as did too many dots at the end\n    (\"http://url.com/withmany.......................................\", {},\n     u('<a href=\"http://url.com/withmany\">http://url.com/withmany</a>.......................................')),\n\n    (\"http://url.com/withmany((((((((((((((((((((((((((((((((((a)\", {},\n     u('<a href=\"http://url.com/withmany\">http://url.com/withmany</a>((((((((((((((((((((((((((((((((((a)')),\n\n    # some examples from http://daringfireball.net/2009/11/liberal_regex_for_matching_urls\n    # plus a fex extras (such as multiple parentheses).\n    (\"http://foo.com/blah_blah\", {},\n     u('<a href=\"http://foo.com/blah_blah\">http://foo.com/blah_blah</a>')),\n\n    (\"http://foo.com/blah_blah/\", {},\n     u('<a href=\"http://foo.com/blah_blah/\">http://foo.com/blah_blah/</a>')),\n\n    (\"(Something like http://foo.com/blah_blah)\", {},\n     u('(Something like <a href=\"http://foo.com/blah_blah\">http://foo.com/blah_blah</a>)')),\n\n    (\"http://foo.com/blah_blah_(wikipedia)\", {},\n     u('<a href=\"http://foo.com/blah_blah_(wikipedia)\">http://foo.com/blah_blah_(wikipedia)</a>')),\n\n    (\"http://foo.com/blah_(blah)_(wikipedia)_blah\", {},\n     u('<a href=\"http://foo.com/blah_(blah)_(wikipedia)_blah\">http://foo.com/blah_(blah)_(wikipedia)_blah</a>')),\n\n    (\"(Something like http://foo.com/blah_blah_(wikipedia))\", {},\n     u('(Something like <a href=\"http://foo.com/blah_blah_(wikipedia)\">http://foo.com/blah_blah_(wikipedia)</a>)')),\n\n    (\"http://foo.com/blah_blah.\", {},\n     u('<a href=\"http://foo.com/blah_blah\">http://foo.com/blah_blah</a>.')),\n\n    (\"http://foo.com/blah_blah/.\", {},\n     u('<a href=\"http://foo.com/blah_blah/\">http://foo.com/blah_blah/</a>.')),\n\n    (\"<http://foo.com/blah_blah>\", {},\n     u('&lt;<a href=\"http://foo.com/blah_blah\">http://foo.com/blah_blah</a>&gt;')),\n\n    (\"<http://foo.com/blah_blah/>\", {},\n     u('&lt;<a href=\"http://foo.com/blah_blah/\">http://foo.com/blah_blah/</a>&gt;')),\n\n    (\"http://foo.com/blah_blah,\", {},\n     u('<a href=\"http://foo.com/blah_blah\">http://foo.com/blah_blah</a>,')),\n\n    (\"http://www.example.com/wpstyle/?p=364.\", {},\n     u('<a href=\"http://www.example.com/wpstyle/?p=364\">http://www.example.com/wpstyle/?p=364</a>.')),\n\n    (\"rdar://1234\",\n     {\"permitted_protocols\": [\"http\", \"rdar\"]},\n     u('<a href=\"rdar://1234\">rdar://1234</a>')),\n\n    (\"rdar:/1234\",\n     {\"permitted_protocols\": [\"rdar\"]},\n     u('<a href=\"rdar:/1234\">rdar:/1234</a>')),\n\n    (\"http://userid:password@example.com:8080\", {},\n     u('<a href=\"http://userid:password@example.com:8080\">http://userid:password@example.com:8080</a>')),\n\n    (\"http://userid@example.com\", {},\n     u('<a href=\"http://userid@example.com\">http://userid@example.com</a>')),\n\n    (\"http://userid@example.com:8080\", {},\n     u('<a href=\"http://userid@example.com:8080\">http://userid@example.com:8080</a>')),\n\n    (\"http://userid:password@example.com\", {},\n     u('<a href=\"http://userid:password@example.com\">http://userid:password@example.com</a>')),\n\n    (\"message://%3c330e7f8409726r6a4ba78dkf1fd71420c1bf6ff@mail.gmail.com%3e\",\n     {\"permitted_protocols\": [\"http\", \"message\"]},\n     u('<a href=\"message://%3c330e7f8409726r6a4ba78dkf1fd71420c1bf6ff@mail.gmail.com%3e\">message://%3c330e7f8409726r6a4ba78dkf1fd71420c1bf6ff@mail.gmail.com%3e</a>')),\n\n    (u(\"http://\\u27a1.ws/\\u4a39\"), {},\n     u('<a href=\"http://\\u27a1.ws/\\u4a39\">http://\\u27a1.ws/\\u4a39</a>')),\n\n    (\"<tag>http://example.com</tag>\", {},\n     u('&lt;tag&gt;<a href=\"http://example.com\">http://example.com</a>&lt;/tag&gt;')),\n\n    (\"Just a www.example.com link.\", {},\n     u('Just a <a href=\"http://www.example.com\">www.example.com</a> link.')),\n\n    (\"Just a www.example.com link.\",\n     {\"require_protocol\": True},\n     u('Just a www.example.com link.')),\n\n    (\"A http://reallylong.com/link/that/exceedsthelenglimit.html\",\n     {\"require_protocol\": True, \"shorten\": True},\n     u('A <a href=\"http://reallylong.com/link/that/exceedsthelenglimit.html\" title=\"http://reallylong.com/link/that/exceedsthelenglimit.html\">http://reallylong.com/link...</a>')),\n\n    (\"A http://reallylongdomainnamethatwillbetoolong.com/hi!\",\n     {\"shorten\": True},\n     u('A <a href=\"http://reallylongdomainnamethatwillbetoolong.com/hi\" title=\"http://reallylongdomainnamethatwillbetoolong.com/hi\">http://reallylongdomainnametha...</a>!')),\n\n    (\"A file:///passwords.txt and http://web.com link\", {},\n     u('A file:///passwords.txt and <a href=\"http://web.com\">http://web.com</a> link')),\n\n    (\"A file:///passwords.txt and http://web.com link\",\n     {\"permitted_protocols\": [\"file\"]},\n     u('A <a href=\"file:///passwords.txt\">file:///passwords.txt</a> and http://web.com link')),\n\n    (\"www.external-link.com\",\n     {\"extra_params\": 'rel=\"nofollow\" class=\"external\"'},\n     u('<a href=\"http://www.external-link.com\" rel=\"nofollow\" class=\"external\">www.external-link.com</a>')),\n\n    (\"www.external-link.com and www.internal-link.com/blogs extra\",\n     {\"extra_params\": lambda href: 'class=\"internal\"' if href.startswith(\"http://www.internal-link.com\") else 'rel=\"nofollow\" class=\"external\"'},\n     u('<a href=\"http://www.external-link.com\" rel=\"nofollow\" class=\"external\">www.external-link.com</a> and <a href=\"http://www.internal-link.com/blogs\" class=\"internal\">www.internal-link.com/blogs</a> extra')),\n\n    (\"www.external-link.com\",\n     {\"extra_params\": lambda href: '    rel=\"nofollow\" class=\"external\"  '},\n     u('<a href=\"http://www.external-link.com\" rel=\"nofollow\" class=\"external\">www.external-link.com</a>')),\n]\n\n\nclass EscapeTestCase(unittest.TestCase):\n    def test_linkify(self):\n        for text, kwargs, html in linkify_tests:\n            linked = tornado.escape.linkify(text, **kwargs)\n            self.assertEqual(linked, html)\n\n    def test_xhtml_escape(self):\n        tests = [\n            (\"<foo>\", \"&lt;foo&gt;\"),\n            (u(\"<foo>\"), u(\"&lt;foo&gt;\")),\n            (b\"<foo>\", b\"&lt;foo&gt;\"),\n\n            (\"<>&\\\"'\", \"&lt;&gt;&amp;&quot;&#39;\"),\n            (\"&amp;\", \"&amp;amp;\"),\n\n            (u(\"<\\u00e9>\"), u(\"&lt;\\u00e9&gt;\")),\n            (b\"<\\xc3\\xa9>\", b\"&lt;\\xc3\\xa9&gt;\"),\n        ]\n        for unescaped, escaped in tests:\n            self.assertEqual(utf8(xhtml_escape(unescaped)), utf8(escaped))\n            self.assertEqual(utf8(unescaped), utf8(xhtml_unescape(escaped)))\n\n    def test_xhtml_unescape_numeric(self):\n        tests = [\n            ('foo&#32;bar', 'foo bar'),\n            ('foo&#x20;bar', 'foo bar'),\n            ('foo&#X20;bar', 'foo bar'),\n            ('foo&#xabc;bar', u('foo\\u0abcbar')),\n            ('foo&#xyz;bar', 'foo&#xyz;bar'),  # invalid encoding\n            ('foo&#;bar', 'foo&#;bar'),        # invalid encoding\n            ('foo&#x;bar', 'foo&#x;bar'),      # invalid encoding\n        ]\n        for escaped, unescaped in tests:\n            self.assertEqual(unescaped, xhtml_unescape(escaped))\n\n    def test_url_escape_unicode(self):\n        tests = [\n            # byte strings are passed through as-is\n            (u('\\u00e9').encode('utf8'), '%C3%A9'),\n            (u('\\u00e9').encode('latin1'), '%E9'),\n\n            # unicode strings become utf8\n            (u('\\u00e9'), '%C3%A9'),\n        ]\n        for unescaped, escaped in tests:\n            self.assertEqual(url_escape(unescaped), escaped)\n\n    def test_url_unescape_unicode(self):\n        tests = [\n            ('%C3%A9', u('\\u00e9'), 'utf8'),\n            ('%C3%A9', u('\\u00c3\\u00a9'), 'latin1'),\n            ('%C3%A9', utf8(u('\\u00e9')), None),\n        ]\n        for escaped, unescaped, encoding in tests:\n            # input strings to url_unescape should only contain ascii\n            # characters, but make sure the function accepts both byte\n            # and unicode strings.\n            self.assertEqual(url_unescape(to_unicode(escaped), encoding), unescaped)\n            self.assertEqual(url_unescape(utf8(escaped), encoding), unescaped)\n\n    def test_url_escape_quote_plus(self):\n        unescaped = '+ #%'\n        plus_escaped = '%2B+%23%25'\n        escaped = '%2B%20%23%25'\n        self.assertEqual(url_escape(unescaped), plus_escaped)\n        self.assertEqual(url_escape(unescaped, plus=False), escaped)\n        self.assertEqual(url_unescape(plus_escaped), unescaped)\n        self.assertEqual(url_unescape(escaped, plus=False), unescaped)\n        self.assertEqual(url_unescape(plus_escaped, encoding=None),\n                         utf8(unescaped))\n        self.assertEqual(url_unescape(escaped, encoding=None, plus=False),\n                         utf8(unescaped))\n\n    def test_escape_return_types(self):\n        # On python2 the escape methods should generally return the same\n        # type as their argument\n        self.assertEqual(type(xhtml_escape(\"foo\")), str)\n        self.assertEqual(type(xhtml_escape(u(\"foo\"))), unicode_type)\n\n    def test_json_decode(self):\n        # json_decode accepts both bytes and unicode, but strings it returns\n        # are always unicode.\n        self.assertEqual(json_decode(b'\"foo\"'), u(\"foo\"))\n        self.assertEqual(json_decode(u('\"foo\"')), u(\"foo\"))\n\n        # Non-ascii bytes are interpreted as utf8\n        self.assertEqual(json_decode(utf8(u('\"\\u00e9\"'))), u(\"\\u00e9\"))\n\n    def test_json_encode(self):\n        # json deals with strings, not bytes.  On python 2 byte strings will\n        # convert automatically if they are utf8; on python 3 byte strings\n        # are not allowed.\n        self.assertEqual(json_decode(json_encode(u(\"\\u00e9\"))), u(\"\\u00e9\"))\n        if bytes is str:\n            self.assertEqual(json_decode(json_encode(utf8(u(\"\\u00e9\")))), u(\"\\u00e9\"))\n            self.assertRaises(UnicodeDecodeError, json_encode, b\"\\xe9\")\n\n    def test_squeeze(self):\n        self.assertEqual(squeeze(u('sequences     of    whitespace   chars')), u('sequences of whitespace chars'))\n\n    def test_recursive_unicode(self):\n        tests = {\n            'dict': {b\"foo\": b\"bar\"},\n            'list': [b\"foo\", b\"bar\"],\n            'tuple': (b\"foo\", b\"bar\"),\n            'bytes': b\"foo\"\n        }\n        self.assertEqual(recursive_unicode(tests['dict']), {u(\"foo\"): u(\"bar\")})\n        self.assertEqual(recursive_unicode(tests['list']), [u(\"foo\"), u(\"bar\")])\n        self.assertEqual(recursive_unicode(tests['tuple']), (u(\"foo\"), u(\"bar\")))\n        self.assertEqual(recursive_unicode(tests['bytes']), u(\"foo\"))\n",
        "#!/usr/bin/env python\nimport unittest\nimport sys\nimport time\nimport traceback\nimport platform\nimport os\n\n\nclass PytestLikeResult(unittest.TestResult):\n    def __init__(self, *args, **kwargs):\n        super().__init__(*args, **kwargs)\n        self.start_time = None\n        # Store (test, outcome, info) for each test:\n        #   outcome ∈ { \"passed\", \"failed\", \"error\", \"skipped\" }\n        #   info can hold the exception or skip reason, etc.\n        self.results = []\n\n    def startTestRun(self):\n        \"\"\"Called once before all tests.\"\"\"\n        self.start_time = time.time()\n        # Print a header similar to pytest\n        print(\"=\" * 29 + \" test session starts \" + \"=\" * 29)\n        # You can customize the platform/versions/paths as you like:\n        print(\n            f\"platform {platform.system().lower()} -- Python {platform.python_version()}\"\n        )\n        print(f\"rootdir: {os.path.abspath(os.curdir)}\")\n\n    def stopTestRun(self):\n        \"\"\"Called once after all tests are done.\"\"\"\n        elapsed = time.time() - self.start_time\n        print()  # blank line after final dot\n\n        # Summarize\n        ntests = len(self.results)\n        print(f\"collected {ntests} items\")\n\n        # Show details of failures/errors\n        failures = [r for r in self.results if r[1] == \"failed\"]\n        errors = [r for r in self.results if r[1] == \"error\"]\n\n        if failures:\n            print(\"\\n\" + \"=\" * 35 + \" FAILURES \" + \"=\" * 35)\n            for test, outcome, err_info in failures:\n                test = test.split(\"::\")\n                if len(test) == 3:\n                    test = f\"{test[1]}.{test[2]}\"\n                else:\n                    test = f\"{test[1]}\"\n                print(f\"__________ {test} __________\")\n                self._print_traceback(err_info)\n        if errors:\n            print(\"\\n\" + \"=\" * 35 + \" ERRORS \" + \"=\" * 37)\n            for test, outcome, err_info in errors:\n                test = test.split(\"::\")\n                if len(test) == 3:\n                    test = f\"{test[1]}.{test[2]}\"\n                else:\n                    test = f\"{test[1]}\"\n                print(f\"__________ {test} __________\")\n                self._print_traceback(err_info)\n\n        # Optional short summary lines for passes/fails/skips\n        print(\"\\n==================== short test summary info ====================\")\n        for test, outcome, info in self.results:\n            if outcome in (\"passed\", \"failed\", \"error\", \"skipped\"):\n                print(f\"{outcome.upper()} {test}\")\n\n        # Final one-line summary: “1 failed, 1 errors, 10 passed in 2.04s”\n        passed_count = sum(1 for r in self.results if r[1] == \"passed\")\n        failed_count = len(failures)\n        error_count = len(errors)\n        skipped_count = sum(1 for r in self.results if r[1] == \"skipped\")\n\n        summary_bits = []\n        if failed_count:\n            summary_bits.append(f\"{failed_count} failed\")\n        if error_count:\n            summary_bits.append(f\"{error_count} error{'s' if error_count != 1 else ''}\")\n        if passed_count:\n            summary_bits.append(f\"{passed_count} passed\")\n        if skipped_count:\n            summary_bits.append(f\"{skipped_count} skipped\")\n\n        summary_str = \", \".join(summary_bits) if summary_bits else \"no tests run\"\n        print(\n            f\"=================== {summary_str} in {elapsed:.2f}s ===================\"\n        )\n\n    def startTest(self, test):\n        \"\"\"Called right before each test method.\"\"\"\n        super().startTest(test)\n        # Print a dot or some indicator\n        print(\".\", end=\"\", flush=True)\n\n    def addSuccess(self, test):\n        super().addSuccess(test)\n        self.results.append((self._test_id(test), \"passed\", None))\n\n    def addFailure(self, test, err):\n        super().addFailure(test, err)\n        self.results.append((self._test_id(test), \"failed\", err))\n\n    def addError(self, test, err):\n        super().addError(test, err)\n        self.results.append((self._test_id(test), \"error\", err))\n\n    def addSkip(self, test, reason):\n        super().addSkip(test, reason)\n        self.results.append((self._test_id(test), \"skipped\", reason))\n\n    def _print_traceback(self, err_info):\n        \"\"\"Helper to print the traceback like pytest does.\"\"\"\n        if isinstance(err_info, tuple) and len(err_info) == 3:\n            exc_type, exc_value, tb = err_info\n            traceback.print_exception(exc_type, exc_value, tb, file=sys.stdout)\n        else:\n            print(str(err_info))\n\n    def _test_id(self, test):\n        try:\n            return f\"{test.__class__.__module__}::{test.__class__.__name__}::{test._testMethodName}\"\n        except AttributeError:\n            # Fallback for _ErrorHolder objects\n            return f\"{test.__class__.__module__}::{test.__class__.__name__}\"\n\n\nclass PytestLikeRunner(unittest.TextTestRunner):\n    \"\"\"A custom TextTestRunner that uses the PytestLikeResult.\"\"\"\n\n    resultclass = PytestLikeResult\n\n    def run(self, test):\n        result = self._makeResult()\n        # Overriding run to ensure we call startTestRun/stopTestRun\n        result.startTestRun()\n        test(result)\n        result.stopTestRun()\n        return result\n\n    def tearDown(self):\n        # Reset signal handlers\n        from twisted.internet import reactor\n\n        reactor.removeAll()\n        reactor.stop()\n        super().tearDown()\n\n\ndef main():\n    loader = unittest.TestLoader()\n    suite = loader.discover(\".\", pattern=\"test_*.py\")  # discover tests in r2e_tests/\n    runner = PytestLikeRunner(verbosity=0)\n    result = runner.run(suite)\n    # Exit with code 0 if all tests passed/skipped, 1 if there were fails/errors\n    sys.exit(not result.wasSuccessful())\n\n\nif __name__ == \"__main__\":\n    main()\n"
    ],
    "test_file_names": [
        "test_1.py",
        "tornado_unittest_runner.py"
    ],
    "setup_res_code": 0,
    "setup_res_stdout": "",
    "setup_res_stderr": "Using CPython 3.9.21\nCreating virtual environment at: .venv\nActivate with: source .venv/bin/activate\nResolved 1 package in 832ms\nPrepared 1 package in 1.15s\nInstalled 1 package in 1ms\n + tornado==4.2.dev1 (from file:///home/gcpuser/buckets/local_repoeval_bucket/repos/tornado_598ffdace41c4ed96eceb98cfcd27388d2f11657)\nResolved 29 packages in 18ms\nInstalled 29 packages in 113ms\n + attrs==24.3.0\n + automat==24.8.1\n + cachetools==5.5.0\n + cffi==1.17.1\n + chardet==5.2.0\n + colorama==0.4.6\n + constantly==23.10.4\n + distlib==0.3.9\n + exceptiongroup==1.2.2\n + filelock==3.16.1\n + hyperlink==21.0.0\n + idna==3.10\n + incremental==24.7.2\n + iniconfig==2.0.0\n + packaging==24.2\n + platformdirs==4.3.6\n + pluggy==1.5.0\n + pycares==4.5.0\n + pycparser==2.22\n + pycurl==7.45.4\n + pyproject-api==1.8.0\n + pytest==8.3.4\n + setuptools==75.6.0\n + tomli==2.2.1\n + tox==4.23.2\n + twisted==24.11.0\n + typing-extensions==4.12.2\n + virtualenv==20.28.0\n + zope-interface==7.2\n",
    "new_commit_res_code": 0,
    "new_commit_res_stdout": "============================= test session starts =============================\nplatform linux -- Python 3.9.21\nrootdir: /home/gcpuser/buckets/local_repoeval_bucket/repos/tornado_598ffdace41c4ed96eceb98cfcd27388d2f11657\n...........\ncollected 11 items\n\n==================== short test summary info ====================\nPASSED r2e_tests.test_1::EscapeTestCase::test_escape_return_types\nPASSED r2e_tests.test_1::EscapeTestCase::test_json_decode\nPASSED r2e_tests.test_1::EscapeTestCase::test_json_encode\nPASSED r2e_tests.test_1::EscapeTestCase::test_linkify\nPASSED r2e_tests.test_1::EscapeTestCase::test_recursive_unicode\nPASSED r2e_tests.test_1::EscapeTestCase::test_squeeze\nPASSED r2e_tests.test_1::EscapeTestCase::test_url_escape_quote_plus\nPASSED r2e_tests.test_1::EscapeTestCase::test_url_escape_unicode\nPASSED r2e_tests.test_1::EscapeTestCase::test_url_unescape_unicode\nPASSED r2e_tests.test_1::EscapeTestCase::test_xhtml_escape\nPASSED r2e_tests.test_1::EscapeTestCase::test_xhtml_unescape_numeric\n=================== 11 passed in 0.00s ===================\n",
    "new_commit_res_stderr": "",
    "old_commit_res_code": 1,
    "old_commit_res_stdout": "============================= test session starts =============================\nplatform linux -- Python 3.9.21\nrootdir: /home/gcpuser/buckets/local_repoeval_bucket/repos/tornado_598ffdace41c4ed96eceb98cfcd27388d2f11657\n...........\ncollected 11 items\n\n=================================== FAILURES ===================================\n__________ EscapeTestCase.test_xhtml_unescape_numeric __________\nTraceback (most recent call last):\n  File \"/home/gcpuser/.local/share/uv/python/cpython-3.9.21-linux-x86_64-gnu/lib/python3.9/unittest/case.py\", line 59, in testPartExecutor\n    yield\n  File \"/home/gcpuser/.local/share/uv/python/cpython-3.9.21-linux-x86_64-gnu/lib/python3.9/unittest/case.py\", line 592, in run\n    self._callTestMethod(testMethod)\n  File \"/home/gcpuser/.local/share/uv/python/cpython-3.9.21-linux-x86_64-gnu/lib/python3.9/unittest/case.py\", line 550, in _callTestMethod\n    method()\n  File \"/home/gcpuser/buckets/local_repoeval_bucket/repos/tornado_598ffdace41c4ed96eceb98cfcd27388d2f11657/r2e_tests/test_1.py\", line 168, in test_xhtml_unescape_numeric\n    self.assertEqual(unescaped, xhtml_unescape(escaped))\nAssertionError: 'foo bar' != 'foo&#x20;bar'\n- foo bar\n+ foo&#x20;bar\n\n\n==================== short test summary info ====================\nPASSED r2e_tests.test_1::EscapeTestCase::test_escape_return_types\nPASSED r2e_tests.test_1::EscapeTestCase::test_json_decode\nPASSED r2e_tests.test_1::EscapeTestCase::test_json_encode\nPASSED r2e_tests.test_1::EscapeTestCase::test_linkify\nPASSED r2e_tests.test_1::EscapeTestCase::test_recursive_unicode\nPASSED r2e_tests.test_1::EscapeTestCase::test_squeeze\nPASSED r2e_tests.test_1::EscapeTestCase::test_url_escape_quote_plus\nPASSED r2e_tests.test_1::EscapeTestCase::test_url_escape_unicode\nPASSED r2e_tests.test_1::EscapeTestCase::test_url_unescape_unicode\nPASSED r2e_tests.test_1::EscapeTestCase::test_xhtml_escape\nFAILED r2e_tests.test_1::EscapeTestCase::test_xhtml_unescape_numeric\n=================== 1 failed, 10 passed in 0.00s ===================\n",
    "old_commit_res_stderr": ""
}
tornado/escape.py tornado/test/escape_test.py
We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
**[ISSUE]**

**Title:** `xhtml_unescape` Fails to Decode Hexadecimal Numeric References

**Description:**
The `xhtml_unescape` function correctly decodes decimal numeric references like `&#32;` to a space character. However, it does not handle hexadecimal numeric references such as `&#x20;` or `&#X20;` properly. This results in strings containing hex numeric references not being unescaped as expected.

**Example Code:**
```python
def example_unescape():
    escaped = 'foo&#x20;bar'
    unescaped = xhtml_unescape(escaped)
    print(unescaped)  # Expected output: 'foo bar'
```

**Expected Behavior:**
All valid numeric references, including both decimal (e.g., `&#32;`) and hexadecimal (e.g., `&#x20;`, `&#X20;`), should be correctly decoded to their corresponding characters. In the example above, `&#x20;` should be converted to a space, resulting in `'foo bar'`.

**Actual Behavior:**
Hexadecimal numeric references are not properly decoded, causing the unescaped string to retain the hex reference instead of converting it to the intended character. Using the example code, the output incorrectly remains as `'foo&#x20;bar'` instead of `'foo bar'`.

**[/ISSUE]**
--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.

--- BEGIN FILE ---
```
### tornado/escape.py
#!/usr/bin/env python
#
# Copyright 2009 Facebook
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.

"""Escaping/unescaping methods for HTML, JSON, URLs, and others.

Also includes a few other miscellaneous string manipulation functions that
have crept in over time.
"""

from __future__ import absolute_import, division, print_function, with_statement

import re
import sys

from tornado.util import unicode_type, basestring_type, u

try:
    from urllib.parse import parse_qs as _parse_qs  # py3
except ImportError:
    from urlparse import parse_qs as _parse_qs  # Python 2.6+

try:
    import htmlentitydefs  # py2
except ImportError:
    import html.entities as htmlentitydefs  # py3

try:
    import urllib.parse as urllib_parse  # py3
except ImportError:
    import urllib as urllib_parse  # py2

import json

try:
    unichr
except NameError:
    unichr = chr

_XHTML_ESCAPE_RE = re.compile('[&<>"\']')
_XHTML_ESCAPE_DICT = {'&': '&amp;', '<': '&lt;', '>': '&gt;', '"': '&quot;',
                      '\'': '&#39;'}


def xhtml_escape(value):
    """Escapes a string so it is valid within HTML or XML.

    Escapes the characters ``<``, ``>``, ``"``, ``'``, and ``&``.
    When used in attribute values the escaped strings must be enclosed
    in quotes.

    .. versionchanged:: 3.2

       Added the single quote to the list of escaped characters.
    """
    return _XHTML_ESCAPE_RE.sub(lambda match: _XHTML_ESCAPE_DICT[match.group(0)],
                                to_basestring(value))


def xhtml_unescape(value):
    """Un-escapes an XML-escaped string."""
    return re.sub(r"&(#?)(\w+?);", _convert_entity, _unicode(value))


# The fact that json_encode wraps json.dumps is an implementation detail.
# Please see https://github.com/tornadoweb/tornado/pull/706
# before sending a pull request that adds **kwargs to this function.
def json_encode(value):
    """JSON-encodes the given Python object."""
    # JSON permits but does not require forward slashes to be escaped.
    # This is useful when json data is emitted in a <script> tag
    # in HTML, as it prevents </script> tags from prematurely terminating
    # the javascript.  Some json libraries do this escaping by default,
    # although python's standard library does not, so we do it here.
    # http://stackoverflow.com/questions/1580647/json-why-are-forward-slashes-escaped
    return json.dumps(value).replace("</", "<\\/")


def json_decode(value):
    """Returns Python objects for the given JSON string."""
    return json.loads(to_basestring(value))


def squeeze(value):
    """Replace all sequences of whitespace chars with a single space."""
    return re.sub(r"[\x00-\x20]+", " ", value).strip()


def url_escape(value, plus=True):
    """Returns a URL-encoded version of the given value.

    If ``plus`` is true (the default), spaces will be represented
    as "+" instead of "%20".  This is appropriate for query strings
    but not for the path component of a URL.  Note that this default
    is the reverse of Python's urllib module.

    .. versionadded:: 3.1
        The ``plus`` argument
    """
    quote = urllib_parse.quote_plus if plus else urllib_parse.quote
    return quote(utf8(value))


# python 3 changed things around enough that we need two separate
# implementations of url_unescape.  We also need our own implementation
# of parse_qs since python 3's version insists on decoding everything.
if sys.version_info[0] < 3:
    def url_unescape(value, encoding='utf-8', plus=True):
        """Decodes the given value from a URL.

        The argument may be either a byte or unicode string.

        If encoding is None, the result will be a byte string.  Otherwise,
        the result is a unicode string in the specified encoding.

        If ``plus`` is true (the default), plus signs will be interpreted
        as spaces (literal plus signs must be represented as "%2B").  This
        is appropriate for query strings and form-encoded values but not
        for the path component of a URL.  Note that this default is the
        reverse of Python's urllib module.

        .. versionadded:: 3.1
           The ``plus`` argument
        """
        unquote = (urllib_parse.unquote_plus if plus else urllib_parse.unquote)
        if encoding is None:
            return unquote(utf8(value))
        else:
            return unicode_type(unquote(utf8(value)), encoding)

    parse_qs_bytes = _parse_qs
else:
    def url_unescape(value, encoding='utf-8', plus=True):
        """Decodes the given value from a URL.

        The argument may be either a byte or unicode string.

        If encoding is None, the result will be a byte string.  Otherwise,
        the result is a unicode string in the specified encoding.

        If ``plus`` is true (the default), plus signs will be interpreted
        as spaces (literal plus signs must be represented as "%2B").  This
        is appropriate for query strings and form-encoded values but not
        for the path component of a URL.  Note that this default is the
        reverse of Python's urllib module.

        .. versionadded:: 3.1
           The ``plus`` argument
        """
        if encoding is None:
            if plus:
                # unquote_to_bytes doesn't have a _plus variant
                value = to_basestring(value).replace('+', ' ')
            return urllib_parse.unquote_to_bytes(value)
        else:
            unquote = (urllib_parse.unquote_plus if plus
                       else urllib_parse.unquote)
            return unquote(to_basestring(value), encoding=encoding)

    def parse_qs_bytes(qs, keep_blank_values=False, strict_parsing=False):
        """Parses a query string like urlparse.parse_qs, but returns the
        values as byte strings.

        Keys still become type str (interpreted as latin1 in python3!)
        because it's too painful to keep them as byte strings in
        python3 and in practice they're nearly always ascii anyway.
        """
        # This is gross, but python3 doesn't give us another way.
        # Latin1 is the universal donor of character encodings.
        result = _parse_qs(qs, keep_blank_values, strict_parsing,
                           encoding='latin1', errors='strict')
        encoded = {}
        for k, v in result.items():
            encoded[k] = [i.encode('latin1') for i in v]
        return encoded


_UTF8_TYPES = (bytes, type(None))


def utf8(value):
    """Converts a string argument to a byte string.

    If the argument is already a byte string or None, it is returned unchanged.
    Otherwise it must be a unicode string and is encoded as utf8.
    """
    if isinstance(value, _UTF8_TYPES):
        return value
    if not isinstance(value, unicode_type):
        raise TypeError(
            "Expected bytes, unicode, or None; got %r" % type(value)
        )
    return value.encode("utf-8")

_TO_UNICODE_TYPES = (unicode_type, type(None))


def to_unicode(value):
    """Converts a string argument to a unicode string.

    If the argument is already a unicode string or None, it is returned
    unchanged.  Otherwise it must be a byte string and is decoded as utf8.
    """
    if isinstance(value, _TO_UNICODE_TYPES):
        return value
    if not isinstance(value, bytes):
        raise TypeError(
            "Expected bytes, unicode, or None; got %r" % type(value)
        )
    return value.decode("utf-8")

# to_unicode was previously named _unicode not because it was private,
# but to avoid conflicts with the built-in unicode() function/type
_unicode = to_unicode

# When dealing with the standard library across python 2 and 3 it is
# sometimes useful to have a direct conversion to the native string type
if str is unicode_type:
    native_str = to_unicode
else:
    native_str = utf8

_BASESTRING_TYPES = (basestring_type, type(None))


def to_basestring(value):
    """Converts a string argument to a subclass of basestring.

    In python2, byte and unicode strings are mostly interchangeable,
    so functions that deal with a user-supplied argument in combination
    with ascii string constants can use either and should return the type
    the user supplied.  In python3, the two types are not interchangeable,
    so this method is needed to convert byte strings to unicode.
    """
    if isinstance(value, _BASESTRING_TYPES):
        return value
    if not isinstance(value, bytes):
        raise TypeError(
            "Expected bytes, unicode, or None; got %r" % type(value)
        )
    return value.decode("utf-8")


def recursive_unicode(obj):
    """Walks a simple data structure, converting byte strings to unicode.

    Supports lists, tuples, and dictionaries.
    """
    if isinstance(obj, dict):
        return dict((recursive_unicode(k), recursive_unicode(v)) for (k, v) in obj.items())
    elif isinstance(obj, list):
        return list(recursive_unicode(i) for i in obj)
    elif isinstance(obj, tuple):
        return tuple(recursive_unicode(i) for i in obj)
    elif isinstance(obj, bytes):
        return to_unicode(obj)
    else:
        return obj

# I originally used the regex from
# http://daringfireball.net/2010/07/improved_regex_for_matching_urls
# but it gets all exponential on certain patterns (such as too many trailing
# dots), causing the regex matcher to never return.
# This regex should avoid those problems.
# Use to_unicode instead of tornado.util.u - we don't want backslashes getting
# processed as escapes.
_URL_RE = re.compile(to_unicode(r"""\b((?:([\w-]+):(/{1,3})|www[.])(?:(?:(?:[^\s&()]|&amp;|&quot;)*(?:[^!"#$%&'()*+,.:;<=>?@\[\]^`{|}~\s]))|(?:\((?:[^\s&()]|&amp;|&quot;)*\)))+)"""))


def linkify(text, shorten=False, extra_params="",
            require_protocol=False, permitted_protocols=["http", "https"]):
    """Converts plain text into HTML with links.

    For example: ``linkify("Hello http://tornadoweb.org!")`` would return
    ``Hello <a href="http://tornadoweb.org">http://tornadoweb.org</a>!``

    Parameters:

    * ``shorten``: Long urls will be shortened for display.

    * ``extra_params``: Extra text to include in the link tag, or a callable
        taking the link as an argument and returning the extra text
        e.g. ``linkify(text, extra_params='rel="nofollow" class="external"')``,
        or::

            def extra_params_cb(url):
                if url.startswith("http://example.com"):
                    return 'class="internal"'
                else:
                    return 'class="external" rel="nofollow"'
            linkify(text, extra_params=extra_params_cb)

    * ``require_protocol``: Only linkify urls which include a protocol. If
        this is False, urls such as www.facebook.com will also be linkified.

    * ``permitted_protocols``: List (or set) of protocols which should be
        linkified, e.g. ``linkify(text, permitted_protocols=["http", "ftp",
        "mailto"])``. It is very unsafe to include protocols such as
        ``javascript``.
    """
    if extra_params and not callable(extra_params):
        extra_params = " " + extra_params.strip()

    def make_link(m):
        url = m.group(1)
        proto = m.group(2)
        if require_protocol and not proto:
            return url  # not protocol, no linkify

        if proto and proto not in permitted_protocols:
            return url  # bad protocol, no linkify

        href = m.group(1)
        if not proto:
            href = "http://" + href   # no proto specified, use http

        if callable(extra_params):
            params = " " + extra_params(href).strip()
        else:
            params = extra_params

        # clip long urls. max_len is just an approximation
        max_len = 30
        if shorten and len(url) > max_len:
            before_clip = url
            if proto:
                proto_len = len(proto) + 1 + len(m.group(3) or "")  # +1 for :
            else:
                proto_len = 0

            parts = url[proto_len:].split("/")
            if len(parts) > 1:
                # Grab the whole host part plus the first bit of the path
                # The path is usually not that interesting once shortened
                # (no more slug, etc), so it really just provides a little
                # extra indication of shortening.
                url = url[:proto_len] + parts[0] + "/" + \
                    parts[1][:8].split('?')[0].split('.')[0]

            if len(url) > max_len * 1.5:  # still too long
                url = url[:max_len]

            if url != before_clip:
                amp = url.rfind('&')
                # avoid splitting html char entities
                if amp > max_len - 5:
                    url = url[:amp]
                url += "..."

                if len(url) >= len(before_clip):
                    url = before_clip
                else:
                    # full url is visible on mouse-over (for those who don't
                    # have a status bar, such as Safari by default)
                    params += ' title="%s"' % href

        return u('<a href="%s"%s>%s</a>') % (href, params, url)

    # First HTML-escape so that our strings are all safe.
    # The regex is modified to avoid character entites other than &amp; so
    # that we won't pick up &quot;, etc.
    text = _unicode(xhtml_escape(text))
    return _URL_RE.sub(make_link, text)


def _convert_entity(m):
    if m.group(1) == "#":
        try:
            return unichr(int(m.group(2)))
        except ValueError:
            return "&#%s;" % m.group(2)
    try:
        return _HTML_UNICODE_MAP[m.group(2)]
    except KeyError:
        return "&%s;" % m.group(2)


def _build_unicode_map():
    unicode_map = {}
    for name, value in htmlentitydefs.name2codepoint.items():
        unicode_map[name] = unichr(value)
    return unicode_map

_HTML_UNICODE_MAP = _build_unicode_map()

```
--- END FILE ---

Please first localize the bug based on the issue statement, and then generate *SEARCH/REPLACE* edits to fix the issue.

Every *SEARCH/REPLACE* edit must use this format:
1. The file path
2. The start of search block: <<<<<<< SEARCH
3. A contiguous chunk of lines to search for in the existing source code
4. The dividing line: =======
5. The lines to replace into the source code
6. The end of the replace block: >>>>>>> REPLACE

Here is an example:

```python
### mathweb/flask/app.py
<<<<<<< SEARCH
from flask import Flask
=======
import math
from flask import Flask
>>>>>>> REPLACE
```
</solution>

Please note that the *SEARCH/REPLACE* edit REQUIRES PROPER INDENTATION. If you would like to add the line '        print(x)', you must fully write that out, with all those spaces before the code!
Wrap each *SEARCH/REPLACE* edit in a code block as shown in the example above. If you have multiple *SEARCH/REPLACE* edits, use a separate code block for each one.
**[ISSUE]**

**Title:** `xhtml_unescape` Fails to Decode Hexadecimal Numeric References

**Description:**
The `xhtml_unescape` function correctly decodes decimal numeric references like `&#32;` to a space character. However, it does not handle hexadecimal numeric references such as `&#x20;` or `&#X20;` properly. This results in strings containing hex numeric references not being unescaped as expected.

**Example Code:**
```python
def example_unescape():
    escaped = 'foo&#x20;bar'
    unescaped = xhtml_unescape(escaped)
    print(unescaped)  # Expected output: 'foo bar'
```

**Expected Behavior:**
All valid numeric references, including both decimal (e.g., `&#32;`) and hexadecimal (e.g., `&#x20;`, `&#X20;`), should be correctly decoded to their corresponding characters. In the example above, `&#x20;` should be converted to a space, resulting in `'foo bar'`.

**Actual Behavior:**
Hexadecimal numeric references are not properly decoded, causing the unescaped string to retain the hex reference instead of converting it to the intended character. Using the example code, the output incorrectly remains as `'foo&#x20;bar'` instead of `'foo bar'`.

**[/ISSUE]**
{
    "EscapeTestCase.test_escape_return_types": "PASSED",
    "EscapeTestCase.test_json_decode": "PASSED",
    "EscapeTestCase.test_json_encode": "PASSED",
    "EscapeTestCase.test_linkify": "PASSED",
    "EscapeTestCase.test_recursive_unicode": "PASSED",
    "EscapeTestCase.test_squeeze": "PASSED",
    "EscapeTestCase.test_url_escape_quote_plus": "PASSED",
    "EscapeTestCase.test_url_escape_unicode": "PASSED",
    "EscapeTestCase.test_url_unescape_unicode": "PASSED",
    "EscapeTestCase.test_xhtml_escape": "PASSED",
    "EscapeTestCase.test_xhtml_unescape_numeric": "PASSED"
}
diff --git a/tornado/escape.py b/tornado/escape.py
index 2852cf51..2f04b468 100644
--- a/tornado/escape.py
+++ b/tornado/escape.py
@@ -378,7 +378,10 @@ def linkify(text, shorten=False, extra_params="",
 def _convert_entity(m):
     if m.group(1) == "#":
         try:
-            return unichr(int(m.group(2)))
+            if m.group(2)[:1].lower() == 'x':
+                return unichr(int(m.group(2)[1:], 16))
+            else:
+                return unichr(int(m.group(2)))
         except ValueError:
             return "&#%s;" % m.group(2)
     try:

{"tornado/escape.py": "#!/usr/bin/env python\n#\n# Copyright 2009 Facebook\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\"); you may\n# not use this file except in compliance with the License. You may obtain\n# a copy of the License at\n#\n#     http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT\n# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the\n# License for the specific language governing permissions and limitations\n# under the License.\n\n\"\"\"Escaping/unescaping methods for HTML, JSON, URLs, and others.\n\nAlso includes a few other miscellaneous string manipulation functions that\nhave crept in over time.\n\"\"\"\n\nfrom __future__ import absolute_import, division, print_function, with_statement\n\nimport re\nimport sys\n\nfrom tornado.util import unicode_type, basestring_type, u\n\ntry:\n    from urllib.parse import parse_qs as _parse_qs  # py3\nexcept ImportError:\n    from urlparse import parse_qs as _parse_qs  # Python 2.6+\n\ntry:\n    import htmlentitydefs  # py2\nexcept ImportError:\n    import html.entities as htmlentitydefs  # py3\n\ntry:\n    import urllib.parse as urllib_parse  # py3\nexcept ImportError:\n    import urllib as urllib_parse  # py2\n\nimport json\n\ntry:\n    unichr\nexcept NameError:\n    unichr = chr\n\n_XHTML_ESCAPE_RE = re.compile('[&<>\"\\']')\n_XHTML_ESCAPE_DICT = {'&': '&amp;', '<': '&lt;', '>': '&gt;', '\"': '&quot;',\n                      '\\'': '&#39;'}\n\n\ndef xhtml_escape(value):\n    \"\"\"Escapes a string so it is valid within HTML or XML.\n\n    Escapes the characters ``<``, ``>``, ``\"``, ``'``, and ``&``.\n    When used in attribute values the escaped strings must be enclosed\n    in quotes.\n\n    .. versionchanged:: 3.2\n\n       Added the single quote to the list of escaped characters.\n    \"\"\"\n    return _XHTML_ESCAPE_RE.sub(lambda match: _XHTML_ESCAPE_DICT[match.group(0)],\n                                to_basestring(value))\n\n\ndef xhtml_unescape(value):\n    \"\"\"Un-escapes an XML-escaped string.\"\"\"\n    return re.sub(r\"&(#?)(\\w+?);\", _convert_entity, _unicode(value))\n\n\n# The fact that json_encode wraps json.dumps is an implementation detail.\n# Please see https://github.com/tornadoweb/tornado/pull/706\n# before sending a pull request that adds **kwargs to this function.\ndef json_encode(value):\n    \"\"\"JSON-encodes the given Python object.\"\"\"\n    # JSON permits but does not require forward slashes to be escaped.\n    # This is useful when json data is emitted in a <script> tag\n    # in HTML, as it prevents </script> tags from prematurely terminating\n    # the javascript.  Some json libraries do this escaping by default,\n    # although python's standard library does not, so we do it here.\n    # http://stackoverflow.com/questions/1580647/json-why-are-forward-slashes-escaped\n    return json.dumps(value).replace(\"</\", \"<\\\\/\")\n\n\ndef json_decode(value):\n    \"\"\"Returns Python objects for the given JSON string.\"\"\"\n    return json.loads(to_basestring(value))\n\n\ndef squeeze(value):\n    \"\"\"Replace all sequences of whitespace chars with a single space.\"\"\"\n    return re.sub(r\"[\\x00-\\x20]+\", \" \", value).strip()\n\n\ndef url_escape(value, plus=True):\n    \"\"\"Returns a URL-encoded version of the given value.\n\n    If ``plus`` is true (the default), spaces will be represented\n    as \"+\" instead of \"%20\".  This is appropriate for query strings\n    but not for the path component of a URL.  Note that this default\n    is the reverse of Python's urllib module.\n\n    .. versionadded:: 3.1\n        The ``plus`` argument\n    \"\"\"\n    quote = urllib_parse.quote_plus if plus else urllib_parse.quote\n    return quote(utf8(value))\n\n\n# python 3 changed things around enough that we need two separate\n# implementations of url_unescape.  We also need our own implementation\n# of parse_qs since python 3's version insists on decoding everything.\nif sys.version_info[0] < 3:\n    def url_unescape(value, encoding='utf-8', plus=True):\n        \"\"\"Decodes the given value from a URL.\n\n        The argument may be either a byte or unicode string.\n\n        If encoding is None, the result will be a byte string.  Otherwise,\n        the result is a unicode string in the specified encoding.\n\n        If ``plus`` is true (the default), plus signs will be interpreted\n        as spaces (literal plus signs must be represented as \"%2B\").  This\n        is appropriate for query strings and form-encoded values but not\n        for the path component of a URL.  Note that this default is the\n        reverse of Python's urllib module.\n\n        .. versionadded:: 3.1\n           The ``plus`` argument\n        \"\"\"\n        unquote = (urllib_parse.unquote_plus if plus else urllib_parse.unquote)\n        if encoding is None:\n            return unquote(utf8(value))\n        else:\n            return unicode_type(unquote(utf8(value)), encoding)\n\n    parse_qs_bytes = _parse_qs\nelse:\n    def url_unescape(value, encoding='utf-8', plus=True):\n        \"\"\"Decodes the given value from a URL.\n\n        The argument may be either a byte or unicode string.\n\n        If encoding is None, the result will be a byte string.  Otherwise,\n        the result is a unicode string in the specified encoding.\n\n        If ``plus`` is true (the default), plus signs will be interpreted\n        as spaces (literal plus signs must be represented as \"%2B\").  This\n        is appropriate for query strings and form-encoded values but not\n        for the path component of a URL.  Note that this default is the\n        reverse of Python's urllib module.\n\n        .. versionadded:: 3.1\n           The ``plus`` argument\n        \"\"\"\n        if encoding is None:\n            if plus:\n                # unquote_to_bytes doesn't have a _plus variant\n                value = to_basestring(value).replace('+', ' ')\n            return urllib_parse.unquote_to_bytes(value)\n        else:\n            unquote = (urllib_parse.unquote_plus if plus\n                       else urllib_parse.unquote)\n            return unquote(to_basestring(value), encoding=encoding)\n\n    def parse_qs_bytes(qs, keep_blank_values=False, strict_parsing=False):\n        \"\"\"Parses a query string like urlparse.parse_qs, but returns the\n        values as byte strings.\n\n        Keys still become type str (interpreted as latin1 in python3!)\n        because it's too painful to keep them as byte strings in\n        python3 and in practice they're nearly always ascii anyway.\n        \"\"\"\n        # This is gross, but python3 doesn't give us another way.\n        # Latin1 is the universal donor of character encodings.\n        result = _parse_qs(qs, keep_blank_values, strict_parsing,\n                           encoding='latin1', errors='strict')\n        encoded = {}\n        for k, v in result.items():\n            encoded[k] = [i.encode('latin1') for i in v]\n        return encoded\n\n\n_UTF8_TYPES = (bytes, type(None))\n\n\ndef utf8(value):\n    \"\"\"Converts a string argument to a byte string.\n\n    If the argument is already a byte string or None, it is returned unchanged.\n    Otherwise it must be a unicode string and is encoded as utf8.\n    \"\"\"\n    if isinstance(value, _UTF8_TYPES):\n        return value\n    if not isinstance(value, unicode_type):\n        raise TypeError(\n            \"Expected bytes, unicode, or None; got %r\" % type(value)\n        )\n    return value.encode(\"utf-8\")\n\n_TO_UNICODE_TYPES = (unicode_type, type(None))\n\n\ndef to_unicode(value):\n    \"\"\"Converts a string argument to a unicode string.\n\n    If the argument is already a unicode string or None, it is returned\n    unchanged.  Otherwise it must be a byte string and is decoded as utf8.\n    \"\"\"\n    if isinstance(value, _TO_UNICODE_TYPES):\n        return value\n    if not isinstance(value, bytes):\n        raise TypeError(\n            \"Expected bytes, unicode, or None; got %r\" % type(value)\n        )\n    return value.decode(\"utf-8\")\n\n# to_unicode was previously named _unicode not because it was private,\n# but to avoid conflicts with the built-in unicode() function/type\n_unicode = to_unicode\n\n# When dealing with the standard library across python 2 and 3 it is\n# sometimes useful to have a direct conversion to the native string type\nif str is unicode_type:\n    native_str = to_unicode\nelse:\n    native_str = utf8\n\n_BASESTRING_TYPES = (basestring_type, type(None))\n\n\ndef to_basestring(value):\n    \"\"\"Converts a string argument to a subclass of basestring.\n\n    In python2, byte and unicode strings are mostly interchangeable,\n    so functions that deal with a user-supplied argument in combination\n    with ascii string constants can use either and should return the type\n    the user supplied.  In python3, the two types are not interchangeable,\n    so this method is needed to convert byte strings to unicode.\n    \"\"\"\n    if isinstance(value, _BASESTRING_TYPES):\n        return value\n    if not isinstance(value, bytes):\n        raise TypeError(\n            \"Expected bytes, unicode, or None; got %r\" % type(value)\n        )\n    return value.decode(\"utf-8\")\n\n\ndef recursive_unicode(obj):\n    \"\"\"Walks a simple data structure, converting byte strings to unicode.\n\n    Supports lists, tuples, and dictionaries.\n    \"\"\"\n    if isinstance(obj, dict):\n        return dict((recursive_unicode(k), recursive_unicode(v)) for (k, v) in obj.items())\n    elif isinstance(obj, list):\n        return list(recursive_unicode(i) for i in obj)\n    elif isinstance(obj, tuple):\n        return tuple(recursive_unicode(i) for i in obj)\n    elif isinstance(obj, bytes):\n        return to_unicode(obj)\n    else:\n        return obj\n\n# I originally used the regex from\n# http://daringfireball.net/2010/07/improved_regex_for_matching_urls\n# but it gets all exponential on certain patterns (such as too many trailing\n# dots), causing the regex matcher to never return.\n# This regex should avoid those problems.\n# Use to_unicode instead of tornado.util.u - we don't want backslashes getting\n# processed as escapes.\n_URL_RE = re.compile(to_unicode(r\"\"\"\\b((?:([\\w-]+):(/{1,3})|www[.])(?:(?:(?:[^\\s&()]|&amp;|&quot;)*(?:[^!\"#$%&'()*+,.:;<=>?@\\[\\]^`{|}~\\s]))|(?:\\((?:[^\\s&()]|&amp;|&quot;)*\\)))+)\"\"\"))\n\n\ndef linkify(text, shorten=False, extra_params=\"\",\n            require_protocol=False, permitted_protocols=[\"http\", \"https\"]):\n    \"\"\"Converts plain text into HTML with links.\n\n    For example: ``linkify(\"Hello http://tornadoweb.org!\")`` would return\n    ``Hello <a href=\"http://tornadoweb.org\">http://tornadoweb.org</a>!``\n\n    Parameters:\n\n    * ``shorten``: Long urls will be shortened for display.\n\n    * ``extra_params``: Extra text to include in the link tag, or a callable\n        taking the link as an argument and returning the extra text\n        e.g. ``linkify(text, extra_params='rel=\"nofollow\" class=\"external\"')``,\n        or::\n\n            def extra_params_cb(url):\n                if url.startswith(\"http://example.com\"):\n                    return 'class=\"internal\"'\n                else:\n                    return 'class=\"external\" rel=\"nofollow\"'\n            linkify(text, extra_params=extra_params_cb)\n\n    * ``require_protocol``: Only linkify urls which include a protocol. If\n        this is False, urls such as www.facebook.com will also be linkified.\n\n    * ``permitted_protocols``: List (or set) of protocols which should be\n        linkified, e.g. ``linkify(text, permitted_protocols=[\"http\", \"ftp\",\n        \"mailto\"])``. It is very unsafe to include protocols such as\n        ``javascript``.\n    \"\"\"\n    if extra_params and not callable(extra_params):\n        extra_params = \" \" + extra_params.strip()\n\n    def make_link(m):\n        url = m.group(1)\n        proto = m.group(2)\n        if require_protocol and not proto:\n            return url  # not protocol, no linkify\n\n        if proto and proto not in permitted_protocols:\n            return url  # bad protocol, no linkify\n\n        href = m.group(1)\n        if not proto:\n            href = \"http://\" + href   # no proto specified, use http\n\n        if callable(extra_params):\n            params = \" \" + extra_params(href).strip()\n        else:\n            params = extra_params\n\n        # clip long urls. max_len is just an approximation\n        max_len = 30\n        if shorten and len(url) > max_len:\n            before_clip = url\n            if proto:\n                proto_len = len(proto) + 1 + len(m.group(3) or \"\")  # +1 for :\n            else:\n                proto_len = 0\n\n            parts = url[proto_len:].split(\"/\")\n            if len(parts) > 1:\n                # Grab the whole host part plus the first bit of the path\n                # The path is usually not that interesting once shortened\n                # (no more slug, etc), so it really just provides a little\n                # extra indication of shortening.\n                url = url[:proto_len] + parts[0] + \"/\" + \\\n                    parts[1][:8].split('?')[0].split('.')[0]\n\n            if len(url) > max_len * 1.5:  # still too long\n                url = url[:max_len]\n\n            if url != before_clip:\n                amp = url.rfind('&')\n                # avoid splitting html char entities\n                if amp > max_len - 5:\n                    url = url[:amp]\n                url += \"...\"\n\n                if len(url) >= len(before_clip):\n                    url = before_clip\n                else:\n                    # full url is visible on mouse-over (for those who don't\n                    # have a status bar, such as Safari by default)\n                    params += ' title=\"%s\"' % href\n\n        return u('<a href=\"%s\"%s>%s</a>') % (href, params, url)\n\n    # First HTML-escape so that our strings are all safe.\n    # The regex is modified to avoid character entites other than &amp; so\n    # that we won't pick up &quot;, etc.\n    text = _unicode(xhtml_escape(text))\n    return _URL_RE.sub(make_link, text)\n\n\ndef _convert_entity(m):\n    if m.group(1) == \"#\":\n        try:\n            return unichr(int(m.group(2)))\n        except ValueError:\n            return \"&#%s;\" % m.group(2)\n    try:\n        return _HTML_UNICODE_MAP[m.group(2)]\n    except KeyError:\n        return \"&%s;\" % m.group(2)\n\n\ndef _build_unicode_map():\n    unicode_map = {}\n    for name, value in htmlentitydefs.name2codepoint.items():\n        unicode_map[name] = unichr(value)\n    return unicode_map\n\n_HTML_UNICODE_MAP = _build_unicode_map()\n"}
A user will ask you to solve a task. You should first draft your thinking process (inner monologue). Then, generate the solution.

Your response format must follow the template below:
<think>
Your thoughts or/and draft, like working through an exercise on scratch paper. Be as casual and as long as you want until you are confident to generate a correct solution.
</think>
<solution>
Final solution presented to the user.
</solution>
system
We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
**[ISSUE]**

**Title:** `xhtml_unescape` Fails to Decode Hexadecimal Numeric References

**Description:**
The `xhtml_unescape` function correctly decodes decimal numeric references like `&#32;` to a space character. However, it does not handle hexadecimal numeric references such as `&#x20;` or `&#X20;` properly. This results in strings containing hex numeric references not being unescaped as expected.

**Example Code:**
```python
def example_unescape():
    escaped = 'foo&#x20;bar'
    unescaped = xhtml_unescape(escaped)
    print(unescaped)  # Expected output: 'foo bar'
```

**Expected Behavior:**
All valid numeric references, including both decimal (e.g., `&#32;`) and hexadecimal (e.g., `&#x20;`, `&#X20;`), should be correctly decoded to their corresponding characters. In the example above, `&#x20;` should be converted to a space, resulting in `'foo bar'`.

**Actual Behavior:**
Hexadecimal numeric references are not properly decoded, causing the unescaped string to retain the hex reference instead of converting it to the intended character. Using the example code, the output incorrectly remains as `'foo&#x20;bar'` instead of `'foo bar'`.

**[/ISSUE]**
--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.

--- BEGIN FILE ---
```
### tornado/escape.py
#!/usr/bin/env python
#
# Copyright 2009 Facebook
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.

"""Escaping/unescaping methods for HTML, JSON, URLs, and others.

Also includes a few other miscellaneous string manipulation functions that
have crept in over time.
"""

from __future__ import absolute_import, division, print_function, with_statement

import re
import sys

from tornado.util import unicode_type, basestring_type, u

try:
    from urllib.parse import parse_qs as _parse_qs  # py3
except ImportError:
    from urlparse import parse_qs as _parse_qs  # Python 2.6+

try:
    import htmlentitydefs  # py2
except ImportError:
    import html.entities as htmlentitydefs  # py3

try:
    import urllib.parse as urllib_parse  # py3
except ImportError:
    import urllib as urllib_parse  # py2

import json

try:
    unichr
except NameError:
    unichr = chr

_XHTML_ESCAPE_RE = re.compile('[&<>"\']')
_XHTML_ESCAPE_DICT = {'&': '&amp;', '<': '&lt;', '>': '&gt;', '"': '&quot;',
                      '\'': '&#39;'}


def xhtml_escape(value):
    """Escapes a string so it is valid within HTML or XML.

    Escapes the characters ``<``, ``>``, ``"``, ``'``, and ``&``.
    When used in attribute values the escaped strings must be enclosed
    in quotes.

    .. versionchanged:: 3.2

       Added the single quote to the list of escaped characters.
    """
    return _XHTML_ESCAPE_RE.sub(lambda match: _XHTML_ESCAPE_DICT[match.group(0)],
                                to_basestring(value))


def xhtml_unescape(value):
    """Un-escapes an XML-escaped string."""
    return re.sub(r"&(#?)(\w+?);", _convert_entity, _unicode(value))


# The fact that json_encode wraps json.dumps is an implementation detail.
# Please see https://github.com/tornadoweb/tornado/pull/706
# before sending a pull request that adds **kwargs to this function.
def json_encode(value):
    """JSON-encodes the given Python object."""
    # JSON permits but does not require forward slashes to be escaped.
    # This is useful when json data is emitted in a <script> tag
    # in HTML, as it prevents </script> tags from prematurely terminating
    # the javascript.  Some json libraries do this escaping by default,
    # although python's standard library does not, so we do it here.
    # http://stackoverflow.com/questions/1580647/json-why-are-forward-slashes-escaped
    return json.dumps(value).replace("</", "<\\/")


def json_decode(value):
    """Returns Python objects for the given JSON string."""
    return json.loads(to_basestring(value))


def squeeze(value):
    """Replace all sequences of whitespace chars with a single space."""
    return re.sub(r"[\x00-\x20]+", " ", value).strip()


def url_escape(value, plus=True):
    """Returns a URL-encoded version of the given value.

    If ``plus`` is true (the default), spaces will be represented
    as "+" instead of "%20".  This is appropriate for query strings
    but not for the path component of a URL.  Note that this default
    is the reverse of Python's urllib module.

    .. versionadded:: 3.1
        The ``plus`` argument
    """
    quote = urllib_parse.quote_plus if plus else urllib_parse.quote
    return quote(utf8(value))


# python 3 changed things around enough that we need two separate
# implementations of url_unescape.  We also need our own implementation
# of parse_qs since python 3's version insists on decoding everything.
if sys.version_info[0] < 3:
    def url_unescape(value, encoding='utf-8', plus=True):
        """Decodes the given value from a URL.

        The argument may be either a byte or unicode string.

        If encoding is None, the result will be a byte string.  Otherwise,
        the result is a unicode string in the specified encoding.

        If ``plus`` is true (the default), plus signs will be interpreted
        as spaces (literal plus signs must be represented as "%2B").  This
        is appropriate for query strings and form-encoded values but not
        for the path component of a URL.  Note that this default is the
        reverse of Python's urllib module.

        .. versionadded:: 3.1
           The ``plus`` argument
        """
        unquote = (urllib_parse.unquote_plus if plus else urllib_parse.unquote)
        if encoding is None:
            return unquote(utf8(value))
        else:
            return unicode_type(unquote(utf8(value)), encoding)

    parse_qs_bytes = _parse_qs
else:
    def url_unescape(value, encoding='utf-8', plus=True):
        """Decodes the given value from a URL.

        The argument may be either a byte or unicode string.

        If encoding is None, the result will be a byte string.  Otherwise,
        the result is a unicode string in the specified encoding.

        If ``plus`` is true (the default), plus signs will be interpreted
        as spaces (literal plus signs must be represented as "%2B").  This
        is appropriate for query strings and form-encoded values but not
        for the path component of a URL.  Note that this default is the
        reverse of Python's urllib module.

        .. versionadded:: 3.1
           The ``plus`` argument
        """
        if encoding is None:
            if plus:
                # unquote_to_bytes doesn't have a _plus variant
                value = to_basestring(value).replace('+', ' ')
            return urllib_parse.unquote_to_bytes(value)
        else:
            unquote = (urllib_parse.unquote_plus if plus
                       else urllib_parse.unquote)
            return unquote(to_basestring(value), encoding=encoding)

    def parse_qs_bytes(qs, keep_blank_values=False, strict_parsing=False):
        """Parses a query string like urlparse.parse_qs, but returns the
        values as byte strings.

        Keys still become type str (interpreted as latin1 in python3!)
        because it's too painful to keep them as byte strings in
        python3 and in practice they're nearly always ascii anyway.
        """
        # This is gross, but python3 doesn't give us another way.
        # Latin1 is the universal donor of character encodings.
        result = _parse_qs(qs, keep_blank_values, strict_parsing,
                           encoding='latin1', errors='strict')
        encoded = {}
        for k, v in result.items():
            encoded[k] = [i.encode('latin1') for i in v]
        return encoded


_UTF8_TYPES = (bytes, type(None))


def utf8(value):
    """Converts a string argument to a byte string.

    If the argument is already a byte string or None, it is returned unchanged.
    Otherwise it must be a unicode string and is encoded as utf8.
    """
    if isinstance(value, _UTF8_TYPES):
        return value
    if not isinstance(value, unicode_type):
        raise TypeError(
            "Expected bytes, unicode, or None; got %r" % type(value)
        )
    return value.encode("utf-8")

_TO_UNICODE_TYPES = (unicode_type, type(None))


def to_unicode(value):
    """Converts a string argument to a unicode string.

    If the argument is already a unicode string or None, it is returned
    unchanged.  Otherwise it must be a byte string and is decoded as utf8.
    """
    if isinstance(value, _TO_UNICODE_TYPES):
        return value
    if not isinstance(value, bytes):
        raise TypeError(
            "Expected bytes, unicode, or None; got %r" % type(value)
        )
    return value.decode("utf-8")

# to_unicode was previously named _unicode not because it was private,
# but to avoid conflicts with the built-in unicode() function/type
_unicode = to_unicode

# When dealing with the standard library across python 2 and 3 it is
# sometimes useful to have a direct conversion to the native string type
if str is unicode_type:
    native_str = to_unicode
else:
    native_str = utf8

_BASESTRING_TYPES = (basestring_type, type(None))


def to_basestring(value):
    """Converts a string argument to a subclass of basestring.

    In python2, byte and unicode strings are mostly interchangeable,
    so functions that deal with a user-supplied argument in combination
    with ascii string constants can use either and should return the type
    the user supplied.  In python3, the two types are not interchangeable,
    so this method is needed to convert byte strings to unicode.
    """
    if isinstance(value, _BASESTRING_TYPES):
        return value
    if not isinstance(value, bytes):
        raise TypeError(
            "Expected bytes, unicode, or None; got %r" % type(value)
        )
    return value.decode("utf-8")


def recursive_unicode(obj):
    """Walks a simple data structure, converting byte strings to unicode.

    Supports lists, tuples, and dictionaries.
    """
    if isinstance(obj, dict):
        return dict((recursive_unicode(k), recursive_unicode(v)) for (k, v) in obj.items())
    elif isinstance(obj, list):
        return list(recursive_unicode(i) for i in obj)
    elif isinstance(obj, tuple):
        return tuple(recursive_unicode(i) for i in obj)
    elif isinstance(obj, bytes):
        return to_unicode(obj)
    else:
        return obj

# I originally used the regex from
# http://daringfireball.net/2010/07/improved_regex_for_matching_urls
# but it gets all exponential on certain patterns (such as too many trailing
# dots), causing the regex matcher to never return.
# This regex should avoid those problems.
# Use to_unicode instead of tornado.util.u - we don't want backslashes getting
# processed as escapes.
_URL_RE = re.compile(to_unicode(r"""\b((?:([\w-]+):(/{1,3})|www[.])(?:(?:(?:[^\s&()]|&amp;|&quot;)*(?:[^!"#$%&'()*+,.:;<=>?@\[\]^`{|}~\s]))|(?:\((?:[^\s&()]|&amp;|&quot;)*\)))+)"""))


def linkify(text, shorten=False, extra_params="",
            require_protocol=False, permitted_protocols=["http", "https"]):
    """Converts plain text into HTML with links.

    For example: ``linkify("Hello http://tornadoweb.org!")`` would return
    ``Hello <a href="http://tornadoweb.org">http://tornadoweb.org</a>!``

    Parameters:

    * ``shorten``: Long urls will be shortened for display.

    * ``extra_params``: Extra text to include in the link tag, or a callable
        taking the link as an argument and returning the extra text
        e.g. ``linkify(text, extra_params='rel="nofollow" class="external"')``,
        or::

            def extra_params_cb(url):
                if url.startswith("http://example.com"):
                    return 'class="internal"'
                else:
                    return 'class="external" rel="nofollow"'
            linkify(text, extra_params=extra_params_cb)

    * ``require_protocol``: Only linkify urls which include a protocol. If
        this is False, urls such as www.facebook.com will also be linkified.

    * ``permitted_protocols``: List (or set) of protocols which should be
        linkified, e.g. ``linkify(text, permitted_protocols=["http", "ftp",
        "mailto"])``. It is very unsafe to include protocols such as
        ``javascript``.
    """
    if extra_params and not callable(extra_params):
        extra_params = " " + extra_params.strip()

    def make_link(m):
        url = m.group(1)
        proto = m.group(2)
        if require_protocol and not proto:
            return url  # not protocol, no linkify

        if proto and proto not in permitted_protocols:
            return url  # bad protocol, no linkify

        href = m.group(1)
        if not proto:
            href = "http://" + href   # no proto specified, use http

        if callable(extra_params):
            params = " " + extra_params(href).strip()
        else:
            params = extra_params

        # clip long urls. max_len is just an approximation
        max_len = 30
        if shorten and len(url) > max_len:
            before_clip = url
            if proto:
                proto_len = len(proto) + 1 + len(m.group(3) or "")  # +1 for :
            else:
                proto_len = 0

            parts = url[proto_len:].split("/")
            if len(parts) > 1:
                # Grab the whole host part plus the first bit of the path
                # The path is usually not that interesting once shortened
                # (no more slug, etc), so it really just provides a little
                # extra indication of shortening.
                url = url[:proto_len] + parts[0] + "/" + \
                    parts[1][:8].split('?')[0].split('.')[0]

            if len(url) > max_len * 1.5:  # still too long
                url = url[:max_len]

            if url != before_clip:
                amp = url.rfind('&')
                # avoid splitting html char entities
                if amp > max_len - 5:
                    url = url[:amp]
                url += "..."

                if len(url) >= len(before_clip):
                    url = before_clip
                else:
                    # full url is visible on mouse-over (for those who don't
                    # have a status bar, such as Safari by default)
                    params += ' title="%s"' % href

        return u('<a href="%s"%s>%s</a>') % (href, params, url)

    # First HTML-escape so that our strings are all safe.
    # The regex is modified to avoid character entites other than &amp; so
    # that we won't pick up &quot;, etc.
    text = _unicode(xhtml_escape(text))
    return _URL_RE.sub(make_link, text)


def _convert_entity(m):
    if m.group(1) == "#":
        try:
            return unichr(int(m.group(2)))
        except ValueError:
            return "&#%s;" % m.group(2)
    try:
        return _HTML_UNICODE_MAP[m.group(2)]
    except KeyError:
        return "&%s;" % m.group(2)


def _build_unicode_map():
    unicode_map = {}
    for name, value in htmlentitydefs.name2codepoint.items():
        unicode_map[name] = unichr(value)
    return unicode_map

_HTML_UNICODE_MAP = _build_unicode_map()

```
--- END FILE ---

Please first localize the bug based on the issue statement, and then generate *SEARCH/REPLACE* edits to fix the issue.

Every *SEARCH/REPLACE* edit must use this format:
1. The file path
2. The start of search block: <<<<<<< SEARCH
3. A contiguous chunk of lines to search for in the existing source code
4. The dividing line: =======
5. The lines to replace into the source code
6. The end of the replace block: >>>>>>> REPLACE

Here is an example:

```python
### mathweb/flask/app.py
<<<<<<< SEARCH
from flask import Flask
=======
import math
from flask import Flask
>>>>>>> REPLACE
```
</solution>

Please note that the *SEARCH/REPLACE* edit REQUIRES PROPER INDENTATION. If you would like to add the line '        print(x)', you must fully write that out, with all those spaces before the code!
Wrap each *SEARCH/REPLACE* edit in a code block as shown in the example above. If you have multiple *SEARCH/REPLACE* edits, use a separate code block for each one.
user