We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
color_sentence matches subtokens 
<!-- Briefly summarize the issue. -->

`color_sentence`  technically does subtoken matching instead of checking for equality between the tokenized sentence and the given token.

Is this intended?

# Stack trace

```bash
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AssertionError: ('\x1b[31mI\x1b[0mf you and I were to meet', 'If you and \x1b[31mI\x1b[0m were to meet')
```

# Steps to reproduce

Here's a MWE

```python
from cleanlab.internal.token_classification_utils import color_sentence

sentence = "If you and I were to meet"
word = "I"
output = color_sentence(sentence, word)
expected_output = "If you and \x1b[31mI\x1b[0m were to meet"

assert output == expected_output, (output, expected_output)
```

<!-- Be as detailed as possible here. If possible, include a self-contained
runnable example that demonstrates the issue. Remember to supply any data
necessary to run your example, or construct your example with synthetic data.
This is not strictly required, but the more detailed your bug report, the more
quickly we can help you and fix the bug. -->

# Additional information

- **Cleanlab version**: 31d43707014cb5c217ee1e9b014a206d554f2fd3
- **Operating system**: Linux 4.19.128-microsoft-standard
- **Python version**: Python 3.9.12

<!-- Please include any other information that could be helpful for debugging. -->


## Definition of `color_sentence`

https://github.com/cleanlab/cleanlab/blob/31d43707014cb5c217ee1e9b014a206d554f2fd3/cleanlab/internal/token_classification_utils.py#L144-L164
Give better error messages when a class has no examples in `find_label_issues`
Currently, we produce various internal errors when a class is not represented. We should either support missing classes, or we should produce something like a `ValueError("find_label_issues requires that each class has at least one example")`.

Right now, we produce a variety of internal errors (data-dependent).

This can be seen as a dupe of #41 and #89, though it suggests at least fixing the confusing error messages as a stop-gap solution until we decide whether or not to support classes with no examples. Potentially related to #266.

# Steps to reproduce

## Example 1

```python
import cleanlab
import numpy as np

labels = np.array([0, 0, 1, 1])
pred_probs = np.array([[0.1, 0.7, 0.2], [0.1, 0.8, 0.1], [0.7, 0.2, 0.1], [0.8, 0.1, 0.1]])
issues = cleanlab.filter.find_label_issues(labels, pred_probs, n_jobs=1)
```

Produces the result:

```
Traceback (most recent call last):
  File ".../error.py", line 6, in <module>
    issues = cleanlab.filter.find_label_issues(labels, pred_probs)
  File ".../src/cleanlab/env/lib/python3.9/site-packages/cleanlab/filter.py", line 457, in find_label_issues
    confident_joint, cl_error_indices = compute_confident_joint(
  File ".../src/cleanlab/env/lib/python3.9/site-packages/cleanlab/count.py", line 435, in compute_confident_joint
    pred_probs_bool = pred_probs >= thresholds - 1e-6
ValueError: operands could not be broadcast together with shapes (4,3) (2,)
```

## Example 2

```python
import cleanlab
import numpy as np

labels = np.array([0, 0])
pred_probs = np.array([[0.3, 0.7], [0.2, 0.8]])
issues = cleanlab.filter.find_label_issues(labels, pred_probs, n_jobs=1)
```

Produces the result:

```
Traceback (most recent call last):
  File ".../error.py", line 6, in <module>
    issues = cleanlab.filter.find_label_issues(labels, pred_probs, n_jobs=1)
  File ".../src/cleanlab/env/lib/python3.9/site-packages/cleanlab/filter.py", line 561, in find_label_issues
    label_issues_masks_per_class = [_prune_by_count(k, args) for k in range(K)]
  File ".../src/cleanlab/env/lib/python3.9/site-packages/cleanlab/filter.py", line 561, in <listcomp>
    label_issues_masks_per_class = [_prune_by_count(k, args) for k in range(K)]
  File ".../src/cleanlab/env/lib/python3.9/site-packages/cleanlab/filter.py", line 204, in _prune_by_count
    if label_counts[k] <= min_examples_per_class:  # No prune if not at least min_examples_per_class
IndexError: index 1 is out of bounds for axis 0 with size 1
```

# Additional information

- **Cleanlab version**: 2.0.0 (and also reproduced on master = 5e0b62d79790028c6c69a76c6af28c813a27ace5, though the stack traces and exceptions are slightly different there)
- **Operating system**: macOS 12.1, shouldn't be relevant here
- **Python version**: Python 3.9.12


--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `cleanlab/internal/token_classification_utils.py`
Content:
```
1 import re
2 import string
3 import numpy as np
4 from termcolor import colored
5 from typing import List, Optional, Callable, Tuple
6 
7 
8 def get_sentence(words: List[str]) -> str:
9     """
10     Get sentence formed by a list of words with minor processing for readability
11 
12     Parameters
13     ----------
14     words: List[str]
15         list of word-level tokens
16 
17     Returns
18     ----------
19     sentence: string
20         sentence formed by list of word-level tokens
21 
22     """
23     sentence = ""
24     for word in words:
25         if word not in string.punctuation or word in ["-", "("]:
26             word = " " + word
27         sentence += word
28     sentence = sentence.replace(" '", "'").replace("( ", "(").strip()
29     return sentence
30 
31 
32 def filter_sentence(
33     sentences: List[str],
34     condition: Optional[Callable[[str], bool]] = None,
35 ) -> Tuple[List[str], List[bool]]:
36     """
37     Filter sentence based on some condition, and returns filter mask
38 
39     Parameters
40     ----------
41         sentences: List[str]
42             list of sentences
43 
44         condition: Optional[Callable[[str], bool]]
45             sentence filtering condition
46 
47     Returns
48     ---------
49         sentences: List[str]
50             list of sentences filtered
51 
52         mask: List[bool]
53             boolean mask such that `mask[i] == True` if the i'th sentence is included in the
54             filtered sentence, otherwise `mask[i] == False`
55 
56     """
57     if not condition:
58         condition = lambda sentence: len(sentence) > 1 and "#" not in sentence
59     mask = list(map(condition, sentences))
60     sentences = [sentence for m, sentence in zip(mask, sentences) if m]
61     return sentences, mask
62 
63 
64 def process_token(token: str, replace: List[Tuple[str, str]] = [("#", "")]) -> str:
65     """
66     Replaces special characters in the tokens
67 
68     Parameters
69     ----------
70         token: str
71             token which potentially contains special characters
72 
73         replace: List[Tuple[str, str]]
74             list of tuples `(s1, s2)`, where all occurances of s1 are replaced by s2
75 
76     Returns
77     ---------
78         processed_token: str
79             processed token whose special character has been replaced
80 
81     Note
82     ----
83         Only applies to characters in the original input token.
84     """
85     replace_dict = {re.escape(k): v for (k, v) in replace}
86     pattern = "|".join(replace_dict.keys())
87     compiled_pattern = re.compile(pattern)
88     replacement = lambda match: replace_dict[re.escape(match.group(0))]
89     processed_token = compiled_pattern.sub(replacement, token)
90     return processed_token
91 
92 
93 def mapping(entities: List[int], maps: List[int]) -> List[int]:
94     """
95     Map a list of entities to its corresponding entities
96 
97     Parameters
98     ----------
99         entities: List[int]
100             a list of given entities
101 
102         maps: List[int]
103             a list of mapped entities, such that the i'th indexed token should be mapped to `maps[i]`
104 
105     Returns
106     ---------
107         mapped_entities: List[int]
108             a list of mapped entities
109 
110     Examples
111     --------
112         >>> unique_identities = [0, 1, 2, 3, 4]  # ["O", "B-PER", "I-PER", "B-LOC", "I-LOC"]
113         >>> maps = [0, 1, 1, 2, 2]  # ["O", "PER", "PER", "LOC", "LOC"]
114         >>> mapping(unique_identities, maps)
115         [0, 1, 1, 2, 2]  # ["O", "PER", "PER", "LOC", "LOC"]
116         >>> mapping([0, 0, 4, 4, 3, 4, 0, 2], maps)
117         [0, 0, 2, 2, 2, 2, 0, 1]  # ["O", "O", "LOC", "LOC", "LOC", "LOC", "O", "PER"]
118     """
119     f = lambda x: maps[x]
120     return list(map(f, entities))
121 
122 
123 def merge_probs(probs: np.ndarray, maps: List[int]) -> np.ndarray:
124     """
125     Merges model-predictive probabilities with desired mapping
126 
127     Parameters
128     ----------
129         probs:
130             np.array of shape `(N, K)`, where N is the number of tokens, and K is the number of classes for the model
131 
132         maps: List[int]
133             a list of mapped index, such that the probability of the token being in the i'th class is mapped to the
134             `maps[i]` index. If `maps[i] == -1`, the i'th column of `probs` is ignored. If `np.any(maps == -1)`, the
135             returned probability is re-normalized.
136 
137     Returns
138     ---------
139         probs_merged:
140             np.array of shape `(N, K')`, where K' is the number of new classes. Probablities are merged and
141             re-normalized if necessary.
142 
143     """
144     old_classes = probs.shape[1]
145     map_size = np.max(maps) + 1
146     probs_merged = np.zeros([len(probs), map_size], dtype=probs.dtype.type)
147 
148     for i in range(old_classes):
149         if maps[i] >= 0:
150             probs_merged[:, maps[i]] += probs[:, i]
151     if -1 in maps:
152         row_sums = probs_merged.sum(axis=1)
153         probs_merged /= row_sums[:, np.newaxis]
154     return probs_merged
155 
156 
157 def color_sentence(sentence: str, word: str) -> str:
158     """
159     Searches for a given token in the sentence and returns the sentence where the given token is colored red
160 
161     Parameters
162     ----------
163         sentence: str
164             a sentence where the word is searched
165 
166         word: str
167             keyword to find in `sentence`. Assumes the word exists in token
168 
169     Returns
170     ---------
171         colored_sentence: str
172             `sentence` where the first occurance of the word is colored red, using `termcolor.colored`
173 
174     """
175     start_idx = sentence.index(word)
176     before, after = sentence[:start_idx], sentence[start_idx + len(word) :]
177     return "%s%s%s" % (before, colored(word, "red"), after)
178
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/cleanlab/internal/token_classification_utils.py b/cleanlab/internal/token_classification_utils.py
--- a/cleanlab/internal/token_classification_utils.py
+++ b/cleanlab/internal/token_classification_utils.py
@@ -160,18 +160,22 @@
 
     Parameters
     ----------
-        sentence: str
+        sentence:
             a sentence where the word is searched
 
-        word: str
-            keyword to find in `sentence`. Assumes the word exists in token
-
+        word:
+            keyword to find in `sentence`. Assumes the word exists in the sentence.
     Returns
     ---------
-        colored_sentence: str
-            `sentence` where the first occurance of the word is colored red, using `termcolor.colored`
+        colored_sentence:
+            `sentence` where the every occurance of the word is colored red, using `termcolor.colored`
 
     """
-    start_idx = sentence.index(word)
-    before, after = sentence[:start_idx], sentence[start_idx + len(word) :]
-    return "%s%s%s" % (before, colored(word, "red"), after)
+    colored_word = colored(word, "red")
+    colored_sentence, number_of_substitions = re.subn(
+        r"\b{}\b".format(word), colored_word, sentence
+    )
+    if number_of_substitions == 0:
+        # Use basic string manipulation if regex fails
+        colored_sentence = sentence.replace(word, colored_word)
+    return colored_sentence

{"golden_diff": "diff --git a/cleanlab/internal/token_classification_utils.py b/cleanlab/internal/token_classification_utils.py\n--- a/cleanlab/internal/token_classification_utils.py\n+++ b/cleanlab/internal/token_classification_utils.py\n@@ -160,18 +160,22 @@\n \n     Parameters\n     ----------\n-        sentence: str\n+        sentence:\n             a sentence where the word is searched\n \n-        word: str\n-            keyword to find in `sentence`. Assumes the word exists in token\n-\n+        word:\n+            keyword to find in `sentence`. Assumes the word exists in the sentence.\n     Returns\n     ---------\n-        colored_sentence: str\n-            `sentence` where the first occurance of the word is colored red, using `termcolor.colored`\n+        colored_sentence:\n+            `sentence` where the every occurance of the word is colored red, using `termcolor.colored`\n \n     \"\"\"\n-    start_idx = sentence.index(word)\n-    before, after = sentence[:start_idx], sentence[start_idx + len(word) :]\n-    return \"%s%s%s\" % (before, colored(word, \"red\"), after)\n+    colored_word = colored(word, \"red\")\n+    colored_sentence, number_of_substitions = re.subn(\n+        r\"\\b{}\\b\".format(word), colored_word, sentence\n+    )\n+    if number_of_substitions == 0:\n+        # Use basic string manipulation if regex fails\n+        colored_sentence = sentence.replace(word, colored_word)\n+    return colored_sentence\n", "issue": "color_sentence matches subtokens \n<!-- Briefly summarize the issue. -->\r\n\r\n`color_sentence`  technically does subtoken matching instead of checking for equality between the tokenized sentence and the given token.\r\n\r\nIs this intended?\r\n\r\n# Stack trace\r\n\r\n```bash\r\nTraceback (most recent call last):\r\n  File \"<stdin>\", line 1, in <module>\r\nAssertionError: ('\\x1b[31mI\\x1b[0mf you and I were to meet', 'If you and \\x1b[31mI\\x1b[0m were to meet')\r\n```\r\n\r\n# Steps to reproduce\r\n\r\nHere's a MWE\r\n\r\n```python\r\nfrom cleanlab.internal.token_classification_utils import color_sentence\r\n\r\nsentence = \"If you and I were to meet\"\r\nword = \"I\"\r\noutput = color_sentence(sentence, word)\r\nexpected_output = \"If you and \\x1b[31mI\\x1b[0m were to meet\"\r\n\r\nassert output == expected_output, (output, expected_output)\r\n```\r\n\r\n<!-- Be as detailed as possible here. If possible, include a self-contained\r\nrunnable example that demonstrates the issue. Remember to supply any data\r\nnecessary to run your example, or construct your example with synthetic data.\r\nThis is not strictly required, but the more detailed your bug report, the more\r\nquickly we can help you and fix the bug. -->\r\n\r\n# Additional information\r\n\r\n- **Cleanlab version**: 31d43707014cb5c217ee1e9b014a206d554f2fd3\r\n- **Operating system**: Linux 4.19.128-microsoft-standard\r\n- **Python version**: Python 3.9.12\r\n\r\n<!-- Please include any other information that could be helpful for debugging. -->\r\n\r\n\r\n## Definition of `color_sentence`\r\n\r\nhttps://github.com/cleanlab/cleanlab/blob/31d43707014cb5c217ee1e9b014a206d554f2fd3/cleanlab/internal/token_classification_utils.py#L144-L164\nGive better error messages when a class has no examples in `find_label_issues`\nCurrently, we produce various internal errors when a class is not represented. We should either support missing classes, or we should produce something like a `ValueError(\"find_label_issues requires that each class has at least one example\")`.\r\n\r\nRight now, we produce a variety of internal errors (data-dependent).\r\n\r\nThis can be seen as a dupe of #41 and #89, though it suggests at least fixing the confusing error messages as a stop-gap solution until we decide whether or not to support classes with no examples. Potentially related to #266.\r\n\r\n# Steps to reproduce\r\n\r\n## Example 1\r\n\r\n```python\r\nimport cleanlab\r\nimport numpy as np\r\n\r\nlabels = np.array([0, 0, 1, 1])\r\npred_probs = np.array([[0.1, 0.7, 0.2], [0.1, 0.8, 0.1], [0.7, 0.2, 0.1], [0.8, 0.1, 0.1]])\r\nissues = cleanlab.filter.find_label_issues(labels, pred_probs, n_jobs=1)\r\n```\r\n\r\nProduces the result:\r\n\r\n```\r\nTraceback (most recent call last):\r\n  File \".../error.py\", line 6, in <module>\r\n    issues = cleanlab.filter.find_label_issues(labels, pred_probs)\r\n  File \".../src/cleanlab/env/lib/python3.9/site-packages/cleanlab/filter.py\", line 457, in find_label_issues\r\n    confident_joint, cl_error_indices = compute_confident_joint(\r\n  File \".../src/cleanlab/env/lib/python3.9/site-packages/cleanlab/count.py\", line 435, in compute_confident_joint\r\n    pred_probs_bool = pred_probs >= thresholds - 1e-6\r\nValueError: operands could not be broadcast together with shapes (4,3) (2,)\r\n```\r\n\r\n## Example 2\r\n\r\n```python\r\nimport cleanlab\r\nimport numpy as np\r\n\r\nlabels = np.array([0, 0])\r\npred_probs = np.array([[0.3, 0.7], [0.2, 0.8]])\r\nissues = cleanlab.filter.find_label_issues(labels, pred_probs, n_jobs=1)\r\n```\r\n\r\nProduces the result:\r\n\r\n```\r\nTraceback (most recent call last):\r\n  File \".../error.py\", line 6, in <module>\r\n    issues = cleanlab.filter.find_label_issues(labels, pred_probs, n_jobs=1)\r\n  File \".../src/cleanlab/env/lib/python3.9/site-packages/cleanlab/filter.py\", line 561, in find_label_issues\r\n    label_issues_masks_per_class = [_prune_by_count(k, args) for k in range(K)]\r\n  File \".../src/cleanlab/env/lib/python3.9/site-packages/cleanlab/filter.py\", line 561, in <listcomp>\r\n    label_issues_masks_per_class = [_prune_by_count(k, args) for k in range(K)]\r\n  File \".../src/cleanlab/env/lib/python3.9/site-packages/cleanlab/filter.py\", line 204, in _prune_by_count\r\n    if label_counts[k] <= min_examples_per_class:  # No prune if not at least min_examples_per_class\r\nIndexError: index 1 is out of bounds for axis 0 with size 1\r\n```\r\n\r\n# Additional information\r\n\r\n- **Cleanlab version**: 2.0.0 (and also reproduced on master = 5e0b62d79790028c6c69a76c6af28c813a27ace5, though the stack traces and exceptions are slightly different there)\r\n- **Operating system**: macOS 12.1, shouldn't be relevant here\r\n- **Python version**: Python 3.9.12\r\n\n", "before_files": [{"content": "import re\nimport string\nimport numpy as np\nfrom termcolor import colored\nfrom typing import List, Optional, Callable, Tuple\n\n\ndef get_sentence(words: List[str]) -> str:\n    \"\"\"\n    Get sentence formed by a list of words with minor processing for readability\n\n    Parameters\n    ----------\n    words: List[str]\n        list of word-level tokens\n\n    Returns\n    ----------\n    sentence: string\n        sentence formed by list of word-level tokens\n\n    \"\"\"\n    sentence = \"\"\n    for word in words:\n        if word not in string.punctuation or word in [\"-\", \"(\"]:\n            word = \" \" + word\n        sentence += word\n    sentence = sentence.replace(\" '\", \"'\").replace(\"( \", \"(\").strip()\n    return sentence\n\n\ndef filter_sentence(\n    sentences: List[str],\n    condition: Optional[Callable[[str], bool]] = None,\n) -> Tuple[List[str], List[bool]]:\n    \"\"\"\n    Filter sentence based on some condition, and returns filter mask\n\n    Parameters\n    ----------\n        sentences: List[str]\n            list of sentences\n\n        condition: Optional[Callable[[str], bool]]\n            sentence filtering condition\n\n    Returns\n    ---------\n        sentences: List[str]\n            list of sentences filtered\n\n        mask: List[bool]\n            boolean mask such that `mask[i] == True` if the i'th sentence is included in the\n            filtered sentence, otherwise `mask[i] == False`\n\n    \"\"\"\n    if not condition:\n        condition = lambda sentence: len(sentence) > 1 and \"#\" not in sentence\n    mask = list(map(condition, sentences))\n    sentences = [sentence for m, sentence in zip(mask, sentences) if m]\n    return sentences, mask\n\n\ndef process_token(token: str, replace: List[Tuple[str, str]] = [(\"#\", \"\")]) -> str:\n    \"\"\"\n    Replaces special characters in the tokens\n\n    Parameters\n    ----------\n        token: str\n            token which potentially contains special characters\n\n        replace: List[Tuple[str, str]]\n            list of tuples `(s1, s2)`, where all occurances of s1 are replaced by s2\n\n    Returns\n    ---------\n        processed_token: str\n            processed token whose special character has been replaced\n\n    Note\n    ----\n        Only applies to characters in the original input token.\n    \"\"\"\n    replace_dict = {re.escape(k): v for (k, v) in replace}\n    pattern = \"|\".join(replace_dict.keys())\n    compiled_pattern = re.compile(pattern)\n    replacement = lambda match: replace_dict[re.escape(match.group(0))]\n    processed_token = compiled_pattern.sub(replacement, token)\n    return processed_token\n\n\ndef mapping(entities: List[int], maps: List[int]) -> List[int]:\n    \"\"\"\n    Map a list of entities to its corresponding entities\n\n    Parameters\n    ----------\n        entities: List[int]\n            a list of given entities\n\n        maps: List[int]\n            a list of mapped entities, such that the i'th indexed token should be mapped to `maps[i]`\n\n    Returns\n    ---------\n        mapped_entities: List[int]\n            a list of mapped entities\n\n    Examples\n    --------\n        >>> unique_identities = [0, 1, 2, 3, 4]  # [\"O\", \"B-PER\", \"I-PER\", \"B-LOC\", \"I-LOC\"]\n        >>> maps = [0, 1, 1, 2, 2]  # [\"O\", \"PER\", \"PER\", \"LOC\", \"LOC\"]\n        >>> mapping(unique_identities, maps)\n        [0, 1, 1, 2, 2]  # [\"O\", \"PER\", \"PER\", \"LOC\", \"LOC\"]\n        >>> mapping([0, 0, 4, 4, 3, 4, 0, 2], maps)\n        [0, 0, 2, 2, 2, 2, 0, 1]  # [\"O\", \"O\", \"LOC\", \"LOC\", \"LOC\", \"LOC\", \"O\", \"PER\"]\n    \"\"\"\n    f = lambda x: maps[x]\n    return list(map(f, entities))\n\n\ndef merge_probs(probs: np.ndarray, maps: List[int]) -> np.ndarray:\n    \"\"\"\n    Merges model-predictive probabilities with desired mapping\n\n    Parameters\n    ----------\n        probs:\n            np.array of shape `(N, K)`, where N is the number of tokens, and K is the number of classes for the model\n\n        maps: List[int]\n            a list of mapped index, such that the probability of the token being in the i'th class is mapped to the\n            `maps[i]` index. If `maps[i] == -1`, the i'th column of `probs` is ignored. If `np.any(maps == -1)`, the\n            returned probability is re-normalized.\n\n    Returns\n    ---------\n        probs_merged:\n            np.array of shape `(N, K')`, where K' is the number of new classes. Probablities are merged and\n            re-normalized if necessary.\n\n    \"\"\"\n    old_classes = probs.shape[1]\n    map_size = np.max(maps) + 1\n    probs_merged = np.zeros([len(probs), map_size], dtype=probs.dtype.type)\n\n    for i in range(old_classes):\n        if maps[i] >= 0:\n            probs_merged[:, maps[i]] += probs[:, i]\n    if -1 in maps:\n        row_sums = probs_merged.sum(axis=1)\n        probs_merged /= row_sums[:, np.newaxis]\n    return probs_merged\n\n\ndef color_sentence(sentence: str, word: str) -> str:\n    \"\"\"\n    Searches for a given token in the sentence and returns the sentence where the given token is colored red\n\n    Parameters\n    ----------\n        sentence: str\n            a sentence where the word is searched\n\n        word: str\n            keyword to find in `sentence`. Assumes the word exists in token\n\n    Returns\n    ---------\n        colored_sentence: str\n            `sentence` where the first occurance of the word is colored red, using `termcolor.colored`\n\n    \"\"\"\n    start_idx = sentence.index(word)\n    before, after = sentence[:start_idx], sentence[start_idx + len(word) :]\n    return \"%s%s%s\" % (before, colored(word, \"red\"), after)\n", "path": "cleanlab/internal/token_classification_utils.py"}], "after_files": [{"content": "import re\nimport string\nimport numpy as np\nfrom termcolor import colored\nfrom typing import List, Optional, Callable, Tuple\n\n\ndef get_sentence(words: List[str]) -> str:\n    \"\"\"\n    Get sentence formed by a list of words with minor processing for readability\n\n    Parameters\n    ----------\n    words: List[str]\n        list of word-level tokens\n\n    Returns\n    ----------\n    sentence: string\n        sentence formed by list of word-level tokens\n\n    \"\"\"\n    sentence = \"\"\n    for word in words:\n        if word not in string.punctuation or word in [\"-\", \"(\"]:\n            word = \" \" + word\n        sentence += word\n    sentence = sentence.replace(\" '\", \"'\").replace(\"( \", \"(\").strip()\n    return sentence\n\n\ndef filter_sentence(\n    sentences: List[str],\n    condition: Optional[Callable[[str], bool]] = None,\n) -> Tuple[List[str], List[bool]]:\n    \"\"\"\n    Filter sentence based on some condition, and returns filter mask\n\n    Parameters\n    ----------\n        sentences: List[str]\n            list of sentences\n\n        condition: Optional[Callable[[str], bool]]\n            sentence filtering condition\n\n    Returns\n    ---------\n        sentences: List[str]\n            list of sentences filtered\n\n        mask: List[bool]\n            boolean mask such that `mask[i] == True` if the i'th sentence is included in the\n            filtered sentence, otherwise `mask[i] == False`\n\n    \"\"\"\n    if not condition:\n        condition = lambda sentence: len(sentence) > 1 and \"#\" not in sentence\n    mask = list(map(condition, sentences))\n    sentences = [sentence for m, sentence in zip(mask, sentences) if m]\n    return sentences, mask\n\n\ndef process_token(token: str, replace: List[Tuple[str, str]] = [(\"#\", \"\")]) -> str:\n    \"\"\"\n    Replaces special characters in the tokens\n\n    Parameters\n    ----------\n        token: str\n            token which potentially contains special characters\n\n        replace: List[Tuple[str, str]]\n            list of tuples `(s1, s2)`, where all occurances of s1 are replaced by s2\n\n    Returns\n    ---------\n        processed_token: str\n            processed token whose special character has been replaced\n\n    Note\n    ----\n        Only applies to characters in the original input token.\n    \"\"\"\n    replace_dict = {re.escape(k): v for (k, v) in replace}\n    pattern = \"|\".join(replace_dict.keys())\n    compiled_pattern = re.compile(pattern)\n    replacement = lambda match: replace_dict[re.escape(match.group(0))]\n    processed_token = compiled_pattern.sub(replacement, token)\n    return processed_token\n\n\ndef mapping(entities: List[int], maps: List[int]) -> List[int]:\n    \"\"\"\n    Map a list of entities to its corresponding entities\n\n    Parameters\n    ----------\n        entities: List[int]\n            a list of given entities\n\n        maps: List[int]\n            a list of mapped entities, such that the i'th indexed token should be mapped to `maps[i]`\n\n    Returns\n    ---------\n        mapped_entities: List[int]\n            a list of mapped entities\n\n    Examples\n    --------\n        >>> unique_identities = [0, 1, 2, 3, 4]  # [\"O\", \"B-PER\", \"I-PER\", \"B-LOC\", \"I-LOC\"]\n        >>> maps = [0, 1, 1, 2, 2]  # [\"O\", \"PER\", \"PER\", \"LOC\", \"LOC\"]\n        >>> mapping(unique_identities, maps)\n        [0, 1, 1, 2, 2]  # [\"O\", \"PER\", \"PER\", \"LOC\", \"LOC\"]\n        >>> mapping([0, 0, 4, 4, 3, 4, 0, 2], maps)\n        [0, 0, 2, 2, 2, 2, 0, 1]  # [\"O\", \"O\", \"LOC\", \"LOC\", \"LOC\", \"LOC\", \"O\", \"PER\"]\n    \"\"\"\n    f = lambda x: maps[x]\n    return list(map(f, entities))\n\n\ndef merge_probs(probs: np.ndarray, maps: List[int]) -> np.ndarray:\n    \"\"\"\n    Merges model-predictive probabilities with desired mapping\n\n    Parameters\n    ----------\n        probs:\n            np.array of shape `(N, K)`, where N is the number of tokens, and K is the number of classes for the model\n\n        maps: List[int]\n            a list of mapped index, such that the probability of the token being in the i'th class is mapped to the\n            `maps[i]` index. If `maps[i] == -1`, the i'th column of `probs` is ignored. If `np.any(maps == -1)`, the\n            returned probability is re-normalized.\n\n    Returns\n    ---------\n        probs_merged:\n            np.array of shape `(N, K')`, where K' is the number of new classes. Probablities are merged and\n            re-normalized if necessary.\n\n    \"\"\"\n    old_classes = probs.shape[1]\n    map_size = np.max(maps) + 1\n    probs_merged = np.zeros([len(probs), map_size], dtype=probs.dtype.type)\n\n    for i in range(old_classes):\n        if maps[i] >= 0:\n            probs_merged[:, maps[i]] += probs[:, i]\n    if -1 in maps:\n        row_sums = probs_merged.sum(axis=1)\n        probs_merged /= row_sums[:, np.newaxis]\n    return probs_merged\n\n\ndef color_sentence(sentence: str, word: str) -> str:\n    \"\"\"\n    Searches for a given token in the sentence and returns the sentence where the given token is colored red\n\n    Parameters\n    ----------\n        sentence:\n            a sentence where the word is searched\n\n        word:\n            keyword to find in `sentence`. Assumes the word exists in the sentence.\n    Returns\n    ---------\n        colored_sentence:\n            `sentence` where the every occurance of the word is colored red, using `termcolor.colored`\n\n    \"\"\"\n    colored_word = colored(word, \"red\")\n    colored_sentence, number_of_substitions = re.subn(\n        r\"\\b{}\\b\".format(word), colored_word, sentence\n    )\n    if number_of_substitions == 0:\n        # Use basic string manipulation if regex fails\n        colored_sentence = sentence.replace(word, colored_word)\n    return colored_sentence\n", "path": "cleanlab/internal/token_classification_utils.py"}]}