We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
Unreasonable Query Result
<!--
**IMPORTANT**:

- Use the [Gensim mailing list](https://groups.google.com/forum/#!forum/gensim) to ask general or usage questions. Github issues are only for bug reports.
- Check [Recipes&FAQ](https://github.com/RaRe-Technologies/gensim/wiki/Recipes-&-FAQ) first for common answers.

Github bug reports that do not include relevant information and context will be closed without an answer. Thanks!
-->

#### Problem description

      **The query result seems not correct. The code is self-explained. Thank you!**

#### Steps/code/corpus to reproduce

Include full tracebacks, logs and datasets if necessary. Please keep the examples minimal ("minimal reproducible example").

```
from gensim.summarization.bm25 import BM25, get_bm25_weights


text1 = "A constellation is a group of stars that are considered to form imaginary outlines or meaningful patterns on the celestial sphere."
text2 = "The 88 modern constellations are formally defined regions of the sky together covering the entire celestial sphere."
text = [text1, text2]

corpus = [text1.split(" "), text2.split(" ")]
print(f'corpus: {corpus}')

query = text2.split(" ")

bm25 = BM25(corpus)
scores = bm25.get_scores(query)
scores = [(s, i) for i, s in enumerate(scores)]
scores.sort(key=lambda t: t[0], reverse=True)
print(f'scores:         {scores}')

for s, idx in scores:
  print(f'{s}\t{idx}: {text[idx]}')
```

Output:

```
-0.3601521710456333         0: A constellation is a group of stars that are considered to form imaginary outlines or meaningful patterns on the celestial sphere.
-0.44989406787023367     1: The 88 modern constellations are formally defined regions of the sky together covering the entire celestial sphere.
```

#### Versions

Please provide the output of:

```python
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import numpy; print("NumPy", numpy.__version__)
import scipy; print("SciPy", scipy.__version__)
import gensim; print("gensim", gensim.__version__)
from gensim.models import word2vec;print("FAST_VERSION", word2vec.FAST_VERSION)
```

Output:
```
macOS-10.14.6-x86_64-i386-64bit
Python 3.8.0 (default, Nov  6 2019, 15:49:01)
[Clang 4.0.1 (tags/RELEASE_401/final)]
NumPy 1.17.4
SciPy 1.3.3
gensim 3.8.1
FAST_VERSION 0
```


--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `gensim/summarization/bm25.py`
Content:
```
1 #!/usr/bin/env python
2 # -*- coding: utf-8 -*-
3 #
4 # Licensed under the GNU LGPL v2.1 - http://www.gnu.org/licenses/lgpl.html
5 
6 """This module contains function of computing rank scores for documents in
7 corpus and helper class `BM25` used in calculations. Original algorithm
8 descibed in [1]_, also you may check Wikipedia page [2]_.
9 
10 
11 .. [1] Robertson, Stephen; Zaragoza, Hugo (2009).  The Probabilistic Relevance Framework: BM25 and Beyond,
12        http://www.staff.city.ac.uk/~sb317/papers/foundations_bm25_review.pdf
13 .. [2] Okapi BM25 on Wikipedia, https://en.wikipedia.org/wiki/Okapi_BM25
14 
15 
16 
17 Examples
18 --------
19 
20 .. sourcecode:: pycon
21 
22     >>> from gensim.summarization.bm25 import get_bm25_weights
23     >>> corpus = [
24     ...     ["black", "cat", "white", "cat"],
25     ...     ["cat", "outer", "space"],
26     ...     ["wag", "dog"]
27     ... ]
28     >>> result = get_bm25_weights(corpus, n_jobs=-1)
29 
30 
31 Data:
32 -----
33 .. data:: PARAM_K1 - Free smoothing parameter for BM25.
34 .. data:: PARAM_B - Free smoothing parameter for BM25.
35 .. data:: EPSILON - Constant used for negative idf of document in corpus.
36 
37 """
38 
39 
40 import math
41 from six import iteritems
42 from six.moves import range
43 from functools import partial
44 from multiprocessing import Pool
45 from ..utils import effective_n_jobs
46 
47 PARAM_K1 = 1.5
48 PARAM_B = 0.75
49 EPSILON = 0.25
50 
51 
52 class BM25(object):
53     """Implementation of Best Matching 25 ranking function.
54 
55     Attributes
56     ----------
57     corpus_size : int
58         Size of corpus (number of documents).
59     avgdl : float
60         Average length of document in `corpus`.
61     doc_freqs : list of dicts of int
62         Dictionary with terms frequencies for each document in `corpus`. Words used as keys and frequencies as values.
63     idf : dict
64         Dictionary with inversed documents frequencies for whole `corpus`. Words used as keys and frequencies as values.
65     doc_len : list of int
66         List of document lengths.
67     """
68 
69     def __init__(self, corpus):
70         """
71         Parameters
72         ----------
73         corpus : list of list of str
74             Given corpus.
75 
76         """
77         self.corpus_size = 0
78         self.avgdl = 0
79         self.doc_freqs = []
80         self.idf = {}
81         self.doc_len = []
82         self._initialize(corpus)
83 
84     def _initialize(self, corpus):
85         """Calculates frequencies of terms in documents and in corpus. Also computes inverse document frequencies."""
86         nd = {}  # word -> number of documents with word
87         num_doc = 0
88         for document in corpus:
89             self.corpus_size += 1
90             self.doc_len.append(len(document))
91             num_doc += len(document)
92 
93             frequencies = {}
94             for word in document:
95                 if word not in frequencies:
96                     frequencies[word] = 0
97                 frequencies[word] += 1
98             self.doc_freqs.append(frequencies)
99 
100             for word, freq in iteritems(frequencies):
101                 if word not in nd:
102                     nd[word] = 0
103                 nd[word] += 1
104 
105         self.avgdl = float(num_doc) / self.corpus_size
106         # collect idf sum to calculate an average idf for epsilon value
107         idf_sum = 0
108         # collect words with negative idf to set them a special epsilon value.
109         # idf can be negative if word is contained in more than half of documents
110         negative_idfs = []
111         for word, freq in iteritems(nd):
112             idf = math.log(self.corpus_size - freq + 0.5) - math.log(freq + 0.5)
113             self.idf[word] = idf
114             idf_sum += idf
115             if idf < 0:
116                 negative_idfs.append(word)
117         self.average_idf = float(idf_sum) / len(self.idf)
118 
119         eps = EPSILON * self.average_idf
120         for word in negative_idfs:
121             self.idf[word] = eps
122 
123     def get_score(self, document, index):
124         """Computes BM25 score of given `document` in relation to item of corpus selected by `index`.
125 
126         Parameters
127         ----------
128         document : list of str
129             Document to be scored.
130         index : int
131             Index of document in corpus selected to score with `document`.
132 
133         Returns
134         -------
135         float
136             BM25 score.
137 
138         """
139         score = 0
140         doc_freqs = self.doc_freqs[index]
141         for word in document:
142             if word not in doc_freqs:
143                 continue
144             score += (self.idf[word] * doc_freqs[word] * (PARAM_K1 + 1)
145                       / (doc_freqs[word] + PARAM_K1 * (1 - PARAM_B + PARAM_B * self.doc_len[index] / self.avgdl)))
146         return score
147 
148     def get_scores(self, document):
149         """Computes and returns BM25 scores of given `document` in relation to
150         every item in corpus.
151 
152         Parameters
153         ----------
154         document : list of str
155             Document to be scored.
156 
157         Returns
158         -------
159         list of float
160             BM25 scores.
161 
162         """
163         scores = [self.get_score(document, index) for index in range(self.corpus_size)]
164         return scores
165 
166     def get_scores_bow(self, document):
167         """Computes and returns BM25 scores of given `document` in relation to
168         every item in corpus.
169 
170         Parameters
171         ----------
172         document : list of str
173             Document to be scored.
174 
175         Returns
176         -------
177         list of float
178             BM25 scores.
179 
180         """
181         scores = []
182         for index in range(self.corpus_size):
183             score = self.get_score(document, index)
184             if score > 0:
185                 scores.append((index, score))
186         return scores
187 
188 
189 def _get_scores_bow(bm25, document):
190     """Helper function for retrieving bm25 scores of given `document` in parallel
191     in relation to every item in corpus.
192 
193     Parameters
194     ----------
195     bm25 : BM25 object
196         BM25 object fitted on the corpus where documents are retrieved.
197     document : list of str
198         Document to be scored.
199 
200     Returns
201     -------
202     list of (index, float)
203         BM25 scores in a bag of weights format.
204 
205     """
206     return bm25.get_scores_bow(document)
207 
208 
209 def _get_scores(bm25, document):
210     """Helper function for retrieving bm25 scores of given `document` in parallel
211     in relation to every item in corpus.
212 
213     Parameters
214     ----------
215     bm25 : BM25 object
216         BM25 object fitted on the corpus where documents are retrieved.
217     document : list of str
218         Document to be scored.
219 
220     Returns
221     -------
222     list of float
223         BM25 scores.
224 
225     """
226     return bm25.get_scores(document)
227 
228 
229 def iter_bm25_bow(corpus, n_jobs=1):
230     """Yield BM25 scores (weights) of documents in corpus.
231     Each document has to be weighted with every document in given corpus.
232 
233     Parameters
234     ----------
235     corpus : list of list of str
236         Corpus of documents.
237     n_jobs : int
238         The number of processes to use for computing bm25.
239 
240     Yields
241     -------
242     list of (index, float)
243         BM25 scores in bag of weights format.
244 
245     Examples
246     --------
247     .. sourcecode:: pycon
248 
249         >>> from gensim.summarization.bm25 import iter_bm25_weights
250         >>> corpus = [
251         ...     ["black", "cat", "white", "cat"],
252         ...     ["cat", "outer", "space"],
253         ...     ["wag", "dog"]
254         ... ]
255         >>> result = iter_bm25_weights(corpus, n_jobs=-1)
256 
257     """
258     bm25 = BM25(corpus)
259 
260     n_processes = effective_n_jobs(n_jobs)
261     if n_processes == 1:
262         for doc in corpus:
263             yield bm25.get_scores_bow(doc)
264         return
265 
266     get_score = partial(_get_scores_bow, bm25)
267     pool = Pool(n_processes)
268 
269     for bow in pool.imap(get_score, corpus):
270         yield bow
271     pool.close()
272     pool.join()
273 
274 
275 def get_bm25_weights(corpus, n_jobs=1):
276     """Returns BM25 scores (weights) of documents in corpus.
277     Each document has to be weighted with every document in given corpus.
278 
279     Parameters
280     ----------
281     corpus : list of list of str
282         Corpus of documents.
283     n_jobs : int
284         The number of processes to use for computing bm25.
285 
286     Returns
287     -------
288     list of list of float
289         BM25 scores.
290 
291     Examples
292     --------
293     .. sourcecode:: pycon
294 
295         >>> from gensim.summarization.bm25 import get_bm25_weights
296         >>> corpus = [
297         ...     ["black", "cat", "white", "cat"],
298         ...     ["cat", "outer", "space"],
299         ...     ["wag", "dog"]
300         ... ]
301         >>> result = get_bm25_weights(corpus, n_jobs=-1)
302 
303     """
304     bm25 = BM25(corpus)
305 
306     n_processes = effective_n_jobs(n_jobs)
307     if n_processes == 1:
308         weights = [bm25.get_scores(doc) for doc in corpus]
309         return weights
310 
311     get_score = partial(_get_scores, bm25)
312     pool = Pool(n_processes)
313     weights = pool.map(get_score, corpus)
314     pool.close()
315     pool.join()
316     return weights
317
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/gensim/summarization/bm25.py b/gensim/summarization/bm25.py
--- a/gensim/summarization/bm25.py
+++ b/gensim/summarization/bm25.py
@@ -37,6 +37,7 @@
 """
 
 
+import logging
 import math
 from six import iteritems
 from six.moves import range
@@ -48,6 +49,8 @@
 PARAM_B = 0.75
 EPSILON = 0.25
 
+logger = logging.getLogger(__name__)
+
 
 class BM25(object):
     """Implementation of Best Matching 25 ranking function.
@@ -116,6 +119,13 @@
                 negative_idfs.append(word)
         self.average_idf = float(idf_sum) / len(self.idf)
 
+        if self.average_idf < 0:
+            logger.warning(
+                'Average inverse document frequency is less than zero. Your corpus of {} documents'
+                ' is either too small or it does not originate from natural text. BM25 may produce'
+                ' unintuitive results.'.format(self.corpus_size)
+            )
+
         eps = EPSILON * self.average_idf
         for word in negative_idfs:
             self.idf[word] = eps

{"golden_diff": "diff --git a/gensim/summarization/bm25.py b/gensim/summarization/bm25.py\n--- a/gensim/summarization/bm25.py\n+++ b/gensim/summarization/bm25.py\n@@ -37,6 +37,7 @@\n \"\"\"\n \n \n+import logging\n import math\n from six import iteritems\n from six.moves import range\n@@ -48,6 +49,8 @@\n PARAM_B = 0.75\n EPSILON = 0.25\n \n+logger = logging.getLogger(__name__)\n+\n \n class BM25(object):\n     \"\"\"Implementation of Best Matching 25 ranking function.\n@@ -116,6 +119,13 @@\n                 negative_idfs.append(word)\n         self.average_idf = float(idf_sum) / len(self.idf)\n \n+        if self.average_idf < 0:\n+            logger.warning(\n+                'Average inverse document frequency is less than zero. Your corpus of {} documents'\n+                ' is either too small or it does not originate from natural text. BM25 may produce'\n+                ' unintuitive results.'.format(self.corpus_size)\n+            )\n+\n         eps = EPSILON * self.average_idf\n         for word in negative_idfs:\n             self.idf[word] = eps\n", "issue": "Unreasonable Query Result\n<!--\r\n**IMPORTANT**:\r\n\r\n- Use the [Gensim mailing list](https://groups.google.com/forum/#!forum/gensim) to ask general or usage questions. Github issues are only for bug reports.\r\n- Check [Recipes&FAQ](https://github.com/RaRe-Technologies/gensim/wiki/Recipes-&-FAQ) first for common answers.\r\n\r\nGithub bug reports that do not include relevant information and context will be closed without an answer. Thanks!\r\n-->\r\n\r\n#### Problem description\r\n\r\n      **The query result seems not correct. The code is self-explained. Thank you!**\r\n\r\n#### Steps/code/corpus to reproduce\r\n\r\nInclude full tracebacks, logs and datasets if necessary. Please keep the examples minimal (\"minimal reproducible example\").\r\n\r\n```\r\nfrom gensim.summarization.bm25 import BM25, get_bm25_weights\r\n\r\n\r\ntext1 = \"A constellation is a group of stars that are considered to form imaginary outlines or meaningful patterns on the celestial sphere.\"\r\ntext2 = \"The 88 modern constellations are formally defined regions of the sky together covering the entire celestial sphere.\"\r\ntext = [text1, text2]\r\n\r\ncorpus = [text1.split(\" \"), text2.split(\" \")]\r\nprint(f'corpus: {corpus}')\r\n\r\nquery = text2.split(\" \")\r\n\r\nbm25 = BM25(corpus)\r\nscores = bm25.get_scores(query)\r\nscores = [(s, i) for i, s in enumerate(scores)]\r\nscores.sort(key=lambda t: t[0], reverse=True)\r\nprint(f'scores:         {scores}')\r\n\r\nfor s, idx in scores:\r\n  print(f'{s}\\t{idx}: {text[idx]}')\r\n```\r\n\r\nOutput:\r\n\r\n```\r\n-0.3601521710456333         0: A constellation is a group of stars that are considered to form imaginary outlines or meaningful patterns on the celestial sphere.\r\n-0.44989406787023367     1: The 88 modern constellations are formally defined regions of the sky together covering the entire celestial sphere.\r\n```\r\n\r\n#### Versions\r\n\r\nPlease provide the output of:\r\n\r\n```python\r\nimport platform; print(platform.platform())\r\nimport sys; print(\"Python\", sys.version)\r\nimport numpy; print(\"NumPy\", numpy.__version__)\r\nimport scipy; print(\"SciPy\", scipy.__version__)\r\nimport gensim; print(\"gensim\", gensim.__version__)\r\nfrom gensim.models import word2vec;print(\"FAST_VERSION\", word2vec.FAST_VERSION)\r\n```\r\n\r\nOutput:\r\n```\r\nmacOS-10.14.6-x86_64-i386-64bit\r\nPython 3.8.0 (default, Nov  6 2019, 15:49:01)\r\n[Clang 4.0.1 (tags/RELEASE_401/final)]\r\nNumPy 1.17.4\r\nSciPy 1.3.3\r\ngensim 3.8.1\r\nFAST_VERSION 0\r\n```\r\n\n", "before_files": [{"content": "#!/usr/bin/env python\n# -*- coding: utf-8 -*-\n#\n# Licensed under the GNU LGPL v2.1 - http://www.gnu.org/licenses/lgpl.html\n\n\"\"\"This module contains function of computing rank scores for documents in\ncorpus and helper class `BM25` used in calculations. Original algorithm\ndescibed in [1]_, also you may check Wikipedia page [2]_.\n\n\n.. [1] Robertson, Stephen; Zaragoza, Hugo (2009).  The Probabilistic Relevance Framework: BM25 and Beyond,\n       http://www.staff.city.ac.uk/~sb317/papers/foundations_bm25_review.pdf\n.. [2] Okapi BM25 on Wikipedia, https://en.wikipedia.org/wiki/Okapi_BM25\n\n\n\nExamples\n--------\n\n.. sourcecode:: pycon\n\n    >>> from gensim.summarization.bm25 import get_bm25_weights\n    >>> corpus = [\n    ...     [\"black\", \"cat\", \"white\", \"cat\"],\n    ...     [\"cat\", \"outer\", \"space\"],\n    ...     [\"wag\", \"dog\"]\n    ... ]\n    >>> result = get_bm25_weights(corpus, n_jobs=-1)\n\n\nData:\n-----\n.. data:: PARAM_K1 - Free smoothing parameter for BM25.\n.. data:: PARAM_B - Free smoothing parameter for BM25.\n.. data:: EPSILON - Constant used for negative idf of document in corpus.\n\n\"\"\"\n\n\nimport math\nfrom six import iteritems\nfrom six.moves import range\nfrom functools import partial\nfrom multiprocessing import Pool\nfrom ..utils import effective_n_jobs\n\nPARAM_K1 = 1.5\nPARAM_B = 0.75\nEPSILON = 0.25\n\n\nclass BM25(object):\n    \"\"\"Implementation of Best Matching 25 ranking function.\n\n    Attributes\n    ----------\n    corpus_size : int\n        Size of corpus (number of documents).\n    avgdl : float\n        Average length of document in `corpus`.\n    doc_freqs : list of dicts of int\n        Dictionary with terms frequencies for each document in `corpus`. Words used as keys and frequencies as values.\n    idf : dict\n        Dictionary with inversed documents frequencies for whole `corpus`. Words used as keys and frequencies as values.\n    doc_len : list of int\n        List of document lengths.\n    \"\"\"\n\n    def __init__(self, corpus):\n        \"\"\"\n        Parameters\n        ----------\n        corpus : list of list of str\n            Given corpus.\n\n        \"\"\"\n        self.corpus_size = 0\n        self.avgdl = 0\n        self.doc_freqs = []\n        self.idf = {}\n        self.doc_len = []\n        self._initialize(corpus)\n\n    def _initialize(self, corpus):\n        \"\"\"Calculates frequencies of terms in documents and in corpus. Also computes inverse document frequencies.\"\"\"\n        nd = {}  # word -> number of documents with word\n        num_doc = 0\n        for document in corpus:\n            self.corpus_size += 1\n            self.doc_len.append(len(document))\n            num_doc += len(document)\n\n            frequencies = {}\n            for word in document:\n                if word not in frequencies:\n                    frequencies[word] = 0\n                frequencies[word] += 1\n            self.doc_freqs.append(frequencies)\n\n            for word, freq in iteritems(frequencies):\n                if word not in nd:\n                    nd[word] = 0\n                nd[word] += 1\n\n        self.avgdl = float(num_doc) / self.corpus_size\n        # collect idf sum to calculate an average idf for epsilon value\n        idf_sum = 0\n        # collect words with negative idf to set them a special epsilon value.\n        # idf can be negative if word is contained in more than half of documents\n        negative_idfs = []\n        for word, freq in iteritems(nd):\n            idf = math.log(self.corpus_size - freq + 0.5) - math.log(freq + 0.5)\n            self.idf[word] = idf\n            idf_sum += idf\n            if idf < 0:\n                negative_idfs.append(word)\n        self.average_idf = float(idf_sum) / len(self.idf)\n\n        eps = EPSILON * self.average_idf\n        for word in negative_idfs:\n            self.idf[word] = eps\n\n    def get_score(self, document, index):\n        \"\"\"Computes BM25 score of given `document` in relation to item of corpus selected by `index`.\n\n        Parameters\n        ----------\n        document : list of str\n            Document to be scored.\n        index : int\n            Index of document in corpus selected to score with `document`.\n\n        Returns\n        -------\n        float\n            BM25 score.\n\n        \"\"\"\n        score = 0\n        doc_freqs = self.doc_freqs[index]\n        for word in document:\n            if word not in doc_freqs:\n                continue\n            score += (self.idf[word] * doc_freqs[word] * (PARAM_K1 + 1)\n                      / (doc_freqs[word] + PARAM_K1 * (1 - PARAM_B + PARAM_B * self.doc_len[index] / self.avgdl)))\n        return score\n\n    def get_scores(self, document):\n        \"\"\"Computes and returns BM25 scores of given `document` in relation to\n        every item in corpus.\n\n        Parameters\n        ----------\n        document : list of str\n            Document to be scored.\n\n        Returns\n        -------\n        list of float\n            BM25 scores.\n\n        \"\"\"\n        scores = [self.get_score(document, index) for index in range(self.corpus_size)]\n        return scores\n\n    def get_scores_bow(self, document):\n        \"\"\"Computes and returns BM25 scores of given `document` in relation to\n        every item in corpus.\n\n        Parameters\n        ----------\n        document : list of str\n            Document to be scored.\n\n        Returns\n        -------\n        list of float\n            BM25 scores.\n\n        \"\"\"\n        scores = []\n        for index in range(self.corpus_size):\n            score = self.get_score(document, index)\n            if score > 0:\n                scores.append((index, score))\n        return scores\n\n\ndef _get_scores_bow(bm25, document):\n    \"\"\"Helper function for retrieving bm25 scores of given `document` in parallel\n    in relation to every item in corpus.\n\n    Parameters\n    ----------\n    bm25 : BM25 object\n        BM25 object fitted on the corpus where documents are retrieved.\n    document : list of str\n        Document to be scored.\n\n    Returns\n    -------\n    list of (index, float)\n        BM25 scores in a bag of weights format.\n\n    \"\"\"\n    return bm25.get_scores_bow(document)\n\n\ndef _get_scores(bm25, document):\n    \"\"\"Helper function for retrieving bm25 scores of given `document` in parallel\n    in relation to every item in corpus.\n\n    Parameters\n    ----------\n    bm25 : BM25 object\n        BM25 object fitted on the corpus where documents are retrieved.\n    document : list of str\n        Document to be scored.\n\n    Returns\n    -------\n    list of float\n        BM25 scores.\n\n    \"\"\"\n    return bm25.get_scores(document)\n\n\ndef iter_bm25_bow(corpus, n_jobs=1):\n    \"\"\"Yield BM25 scores (weights) of documents in corpus.\n    Each document has to be weighted with every document in given corpus.\n\n    Parameters\n    ----------\n    corpus : list of list of str\n        Corpus of documents.\n    n_jobs : int\n        The number of processes to use for computing bm25.\n\n    Yields\n    -------\n    list of (index, float)\n        BM25 scores in bag of weights format.\n\n    Examples\n    --------\n    .. sourcecode:: pycon\n\n        >>> from gensim.summarization.bm25 import iter_bm25_weights\n        >>> corpus = [\n        ...     [\"black\", \"cat\", \"white\", \"cat\"],\n        ...     [\"cat\", \"outer\", \"space\"],\n        ...     [\"wag\", \"dog\"]\n        ... ]\n        >>> result = iter_bm25_weights(corpus, n_jobs=-1)\n\n    \"\"\"\n    bm25 = BM25(corpus)\n\n    n_processes = effective_n_jobs(n_jobs)\n    if n_processes == 1:\n        for doc in corpus:\n            yield bm25.get_scores_bow(doc)\n        return\n\n    get_score = partial(_get_scores_bow, bm25)\n    pool = Pool(n_processes)\n\n    for bow in pool.imap(get_score, corpus):\n        yield bow\n    pool.close()\n    pool.join()\n\n\ndef get_bm25_weights(corpus, n_jobs=1):\n    \"\"\"Returns BM25 scores (weights) of documents in corpus.\n    Each document has to be weighted with every document in given corpus.\n\n    Parameters\n    ----------\n    corpus : list of list of str\n        Corpus of documents.\n    n_jobs : int\n        The number of processes to use for computing bm25.\n\n    Returns\n    -------\n    list of list of float\n        BM25 scores.\n\n    Examples\n    --------\n    .. sourcecode:: pycon\n\n        >>> from gensim.summarization.bm25 import get_bm25_weights\n        >>> corpus = [\n        ...     [\"black\", \"cat\", \"white\", \"cat\"],\n        ...     [\"cat\", \"outer\", \"space\"],\n        ...     [\"wag\", \"dog\"]\n        ... ]\n        >>> result = get_bm25_weights(corpus, n_jobs=-1)\n\n    \"\"\"\n    bm25 = BM25(corpus)\n\n    n_processes = effective_n_jobs(n_jobs)\n    if n_processes == 1:\n        weights = [bm25.get_scores(doc) for doc in corpus]\n        return weights\n\n    get_score = partial(_get_scores, bm25)\n    pool = Pool(n_processes)\n    weights = pool.map(get_score, corpus)\n    pool.close()\n    pool.join()\n    return weights\n", "path": "gensim/summarization/bm25.py"}], "after_files": [{"content": "#!/usr/bin/env python\n# -*- coding: utf-8 -*-\n#\n# Licensed under the GNU LGPL v2.1 - http://www.gnu.org/licenses/lgpl.html\n\n\"\"\"This module contains function of computing rank scores for documents in\ncorpus and helper class `BM25` used in calculations. Original algorithm\ndescibed in [1]_, also you may check Wikipedia page [2]_.\n\n\n.. [1] Robertson, Stephen; Zaragoza, Hugo (2009).  The Probabilistic Relevance Framework: BM25 and Beyond,\n       http://www.staff.city.ac.uk/~sb317/papers/foundations_bm25_review.pdf\n.. [2] Okapi BM25 on Wikipedia, https://en.wikipedia.org/wiki/Okapi_BM25\n\n\n\nExamples\n--------\n\n.. sourcecode:: pycon\n\n    >>> from gensim.summarization.bm25 import get_bm25_weights\n    >>> corpus = [\n    ...     [\"black\", \"cat\", \"white\", \"cat\"],\n    ...     [\"cat\", \"outer\", \"space\"],\n    ...     [\"wag\", \"dog\"]\n    ... ]\n    >>> result = get_bm25_weights(corpus, n_jobs=-1)\n\n\nData:\n-----\n.. data:: PARAM_K1 - Free smoothing parameter for BM25.\n.. data:: PARAM_B - Free smoothing parameter for BM25.\n.. data:: EPSILON - Constant used for negative idf of document in corpus.\n\n\"\"\"\n\n\nimport logging\nimport math\nfrom six import iteritems\nfrom six.moves import range\nfrom functools import partial\nfrom multiprocessing import Pool\nfrom ..utils import effective_n_jobs\n\nPARAM_K1 = 1.5\nPARAM_B = 0.75\nEPSILON = 0.25\n\nlogger = logging.getLogger(__name__)\n\n\nclass BM25(object):\n    \"\"\"Implementation of Best Matching 25 ranking function.\n\n    Attributes\n    ----------\n    corpus_size : int\n        Size of corpus (number of documents).\n    avgdl : float\n        Average length of document in `corpus`.\n    doc_freqs : list of dicts of int\n        Dictionary with terms frequencies for each document in `corpus`. Words used as keys and frequencies as values.\n    idf : dict\n        Dictionary with inversed documents frequencies for whole `corpus`. Words used as keys and frequencies as values.\n    doc_len : list of int\n        List of document lengths.\n    \"\"\"\n\n    def __init__(self, corpus):\n        \"\"\"\n        Parameters\n        ----------\n        corpus : list of list of str\n            Given corpus.\n\n        \"\"\"\n        self.corpus_size = 0\n        self.avgdl = 0\n        self.doc_freqs = []\n        self.idf = {}\n        self.doc_len = []\n        self._initialize(corpus)\n\n    def _initialize(self, corpus):\n        \"\"\"Calculates frequencies of terms in documents and in corpus. Also computes inverse document frequencies.\"\"\"\n        nd = {}  # word -> number of documents with word\n        num_doc = 0\n        for document in corpus:\n            self.corpus_size += 1\n            self.doc_len.append(len(document))\n            num_doc += len(document)\n\n            frequencies = {}\n            for word in document:\n                if word not in frequencies:\n                    frequencies[word] = 0\n                frequencies[word] += 1\n            self.doc_freqs.append(frequencies)\n\n            for word, freq in iteritems(frequencies):\n                if word not in nd:\n                    nd[word] = 0\n                nd[word] += 1\n\n        self.avgdl = float(num_doc) / self.corpus_size\n        # collect idf sum to calculate an average idf for epsilon value\n        idf_sum = 0\n        # collect words with negative idf to set them a special epsilon value.\n        # idf can be negative if word is contained in more than half of documents\n        negative_idfs = []\n        for word, freq in iteritems(nd):\n            idf = math.log(self.corpus_size - freq + 0.5) - math.log(freq + 0.5)\n            self.idf[word] = idf\n            idf_sum += idf\n            if idf < 0:\n                negative_idfs.append(word)\n        self.average_idf = float(idf_sum) / len(self.idf)\n\n        if self.average_idf < 0:\n            logger.warning(\n                'Average inverse document frequency is less than zero. Your corpus of {} documents'\n                ' is either too small or it does not originate from natural text. BM25 may produce'\n                ' unintuitive results.'.format(self.corpus_size)\n            )\n\n        eps = EPSILON * self.average_idf\n        for word in negative_idfs:\n            self.idf[word] = eps\n\n    def get_score(self, document, index):\n        \"\"\"Computes BM25 score of given `document` in relation to item of corpus selected by `index`.\n\n        Parameters\n        ----------\n        document : list of str\n            Document to be scored.\n        index : int\n            Index of document in corpus selected to score with `document`.\n\n        Returns\n        -------\n        float\n            BM25 score.\n\n        \"\"\"\n        score = 0\n        doc_freqs = self.doc_freqs[index]\n        for word in document:\n            if word not in doc_freqs:\n                continue\n            score += (self.idf[word] * doc_freqs[word] * (PARAM_K1 + 1)\n                      / (doc_freqs[word] + PARAM_K1 * (1 - PARAM_B + PARAM_B * self.doc_len[index] / self.avgdl)))\n        return score\n\n    def get_scores(self, document):\n        \"\"\"Computes and returns BM25 scores of given `document` in relation to\n        every item in corpus.\n\n        Parameters\n        ----------\n        document : list of str\n            Document to be scored.\n\n        Returns\n        -------\n        list of float\n            BM25 scores.\n\n        \"\"\"\n        scores = [self.get_score(document, index) for index in range(self.corpus_size)]\n        return scores\n\n    def get_scores_bow(self, document):\n        \"\"\"Computes and returns BM25 scores of given `document` in relation to\n        every item in corpus.\n\n        Parameters\n        ----------\n        document : list of str\n            Document to be scored.\n\n        Returns\n        -------\n        list of float\n            BM25 scores.\n\n        \"\"\"\n        scores = []\n        for index in range(self.corpus_size):\n            score = self.get_score(document, index)\n            if score > 0:\n                scores.append((index, score))\n        return scores\n\n\ndef _get_scores_bow(bm25, document):\n    \"\"\"Helper function for retrieving bm25 scores of given `document` in parallel\n    in relation to every item in corpus.\n\n    Parameters\n    ----------\n    bm25 : BM25 object\n        BM25 object fitted on the corpus where documents are retrieved.\n    document : list of str\n        Document to be scored.\n\n    Returns\n    -------\n    list of (index, float)\n        BM25 scores in a bag of weights format.\n\n    \"\"\"\n    return bm25.get_scores_bow(document)\n\n\ndef _get_scores(bm25, document):\n    \"\"\"Helper function for retrieving bm25 scores of given `document` in parallel\n    in relation to every item in corpus.\n\n    Parameters\n    ----------\n    bm25 : BM25 object\n        BM25 object fitted on the corpus where documents are retrieved.\n    document : list of str\n        Document to be scored.\n\n    Returns\n    -------\n    list of float\n        BM25 scores.\n\n    \"\"\"\n    return bm25.get_scores(document)\n\n\ndef iter_bm25_bow(corpus, n_jobs=1):\n    \"\"\"Yield BM25 scores (weights) of documents in corpus.\n    Each document has to be weighted with every document in given corpus.\n\n    Parameters\n    ----------\n    corpus : list of list of str\n        Corpus of documents.\n    n_jobs : int\n        The number of processes to use for computing bm25.\n\n    Yields\n    -------\n    list of (index, float)\n        BM25 scores in bag of weights format.\n\n    Examples\n    --------\n    .. sourcecode:: pycon\n\n        >>> from gensim.summarization.bm25 import iter_bm25_weights\n        >>> corpus = [\n        ...     [\"black\", \"cat\", \"white\", \"cat\"],\n        ...     [\"cat\", \"outer\", \"space\"],\n        ...     [\"wag\", \"dog\"]\n        ... ]\n        >>> result = iter_bm25_weights(corpus, n_jobs=-1)\n\n    \"\"\"\n    bm25 = BM25(corpus)\n\n    n_processes = effective_n_jobs(n_jobs)\n    if n_processes == 1:\n        for doc in corpus:\n            yield bm25.get_scores_bow(doc)\n        return\n\n    get_score = partial(_get_scores_bow, bm25)\n    pool = Pool(n_processes)\n\n    for bow in pool.imap(get_score, corpus):\n        yield bow\n    pool.close()\n    pool.join()\n\n\ndef get_bm25_weights(corpus, n_jobs=1):\n    \"\"\"Returns BM25 scores (weights) of documents in corpus.\n    Each document has to be weighted with every document in given corpus.\n\n    Parameters\n    ----------\n    corpus : list of list of str\n        Corpus of documents.\n    n_jobs : int\n        The number of processes to use for computing bm25.\n\n    Returns\n    -------\n    list of list of float\n        BM25 scores.\n\n    Examples\n    --------\n    .. sourcecode:: pycon\n\n        >>> from gensim.summarization.bm25 import get_bm25_weights\n        >>> corpus = [\n        ...     [\"black\", \"cat\", \"white\", \"cat\"],\n        ...     [\"cat\", \"outer\", \"space\"],\n        ...     [\"wag\", \"dog\"]\n        ... ]\n        >>> result = get_bm25_weights(corpus, n_jobs=-1)\n\n    \"\"\"\n    bm25 = BM25(corpus)\n\n    n_processes = effective_n_jobs(n_jobs)\n    if n_processes == 1:\n        weights = [bm25.get_scores(doc) for doc in corpus]\n        return weights\n\n    get_score = partial(_get_scores, bm25)\n    pool = Pool(n_processes)\n    weights = pool.map(get_score, corpus)\n    pool.close()\n    pool.join()\n    return weights\n", "path": "gensim/summarization/bm25.py"}]}