We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
The Reddit cog occassionally fails to relay new /r/python messages
Occasionally, the [Reddit cog](/python-discord/bot/blob/master/bot/cogs/reddit.py) stops relaying new messages to the #reddit channel. From what I've seen, there are two ways in which the cog breaks and both should be easy to fix.

#### 1. The background task fails on unexpected http responses
The background task to fetch new posts, the `self.new_posts_task` created in the `on_ready` event listener, assumes that the response we get from the Reddit API is valid `json` in [this line](/python-discord/bot/blob/master/bot/cogs/reddit.py#L55). If, for some temporary reason, we don't get a `json` response, but a `text/html` response (404, probably), the task fails with the following exception:

```py
In [160]: bot.get_cog("Reddit").new_posts_task
Out[160]: 
  File "/bot/bot/cogs/eval.py", line 167, in _eval
    res = await func()
  File "<string>", line 8, in func
  File "/bot/bot/cogs/eval.py", line 167, in _eval
    res = await func()
  File "<string>", line 8, in func
  File "/bot/bot/cogs/reddit.py", line 142, in poll_new_posts
    posts = await self.fetch_posts(f"{subreddit}/new")
  File "/bot/bot/cogs/reddit.py", line 55, in fetch_posts
    content = await response.json()
  File "/bot/.venv/lib/python3.6/site-packages/aiohttp/client_reqrep.py", line 938, in json
    headers=self.headers)
aiohttp.client_exceptions.ContentTypeError: 0, message='Attempt to decode JSON with unexpected mimetype: text/html; charset=utf-8'
```

This line is in the `fetch_posts` utility function used by both the `new_posts_task` as well as the `top_weekly_posts_task`. This means that can break because of this error. 

The likely solution is to either handle the exception or check the response status code before trying to parse it as json. We probably want to either have some kind of retry logic in the handling of the non-200 response, since otherwise the weekly top posts task skips a week.

#### ~~2. The channel can't be initialized during `on_ready`~~

~~Similar to the issue we had with the watch channels, it can occur that the bot's internal channel cache isn't fully loaded when the `on_ready` event fires. This means that the channel will not be retrieved and the tasks will never be started. This only happens intermittently, but it does occasionally happen.~~

~~In such cases, the `self.bot.get_channel` returns `None`, because the channel has not been loaded into the internal bot cache yet at that point.~~

```py
    async def on_ready(self):
        self.reddit_channel = self.bot.get_channel(Channels.reddit)
```

~~While this could be fixed by adding a slight delay or a bit of retry logic, another option is to wait for the migration to a later version of `discord.py` and use `await self.bot.fetch_channel` here instead. That will fetch the channel from the API, bypassing any issues we may have with the internal bot channel cache.~~

--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `bot/cogs/reddit.py`
Content:
```
1 import asyncio
2 import logging
3 import random
4 import textwrap
5 from datetime import datetime, timedelta
6 from typing import List
7 
8 from discord import Colour, Embed, Message, TextChannel
9 from discord.ext.commands import Bot, Cog, Context, group
10 
11 from bot.constants import Channels, ERROR_REPLIES, Reddit as RedditConfig, STAFF_ROLES
12 from bot.converters import Subreddit
13 from bot.decorators import with_role
14 from bot.pagination import LinePaginator
15 
16 log = logging.getLogger(__name__)
17 
18 
19 class Reddit(Cog):
20     """Track subreddit posts and show detailed statistics about them."""
21 
22     HEADERS = {"User-Agent": "Discord Bot: PythonDiscord (https://pythondiscord.com/)"}
23     URL = "https://www.reddit.com"
24 
25     def __init__(self, bot: Bot):
26         self.bot = bot
27 
28         self.reddit_channel = None
29 
30         self.prev_lengths = {}
31         self.last_ids = {}
32 
33         self.new_posts_task = None
34         self.top_weekly_posts_task = None
35 
36     async def fetch_posts(self, route: str, *, amount: int = 25, params: dict = None) -> List[dict]:
37         """A helper method to fetch a certain amount of Reddit posts at a given route."""
38         # Reddit's JSON responses only provide 25 posts at most.
39         if not 25 >= amount > 0:
40             raise ValueError("Invalid amount of subreddit posts requested.")
41 
42         if params is None:
43             params = {}
44 
45         response = await self.bot.http_session.get(
46             url=f"{self.URL}/{route}.json",
47             headers=self.HEADERS,
48             params=params
49         )
50 
51         content = await response.json()
52         posts = content["data"]["children"]
53 
54         return posts[:amount]
55 
56     async def send_top_posts(
57         self, channel: TextChannel, subreddit: Subreddit, content: str = None, time: str = "all"
58     ) -> Message:
59         """Create an embed for the top posts, then send it in a given TextChannel."""
60         # Create the new spicy embed.
61         embed = Embed()
62         embed.description = ""
63 
64         # Get the posts
65         posts = await self.fetch_posts(
66             route=f"{subreddit}/top",
67             amount=5,
68             params={
69                 "t": time
70             }
71         )
72 
73         if not posts:
74             embed.title = random.choice(ERROR_REPLIES)
75             embed.colour = Colour.red()
76             embed.description = (
77                 "Sorry! We couldn't find any posts from that subreddit. "
78                 "If this problem persists, please let us know."
79             )
80 
81             return await channel.send(
82                 embed=embed
83             )
84 
85         for post in posts:
86             data = post["data"]
87 
88             text = data["selftext"]
89             if text:
90                 text = textwrap.shorten(text, width=128, placeholder="...")
91                 text += "\n"  # Add newline to separate embed info
92 
93             ups = data["ups"]
94             comments = data["num_comments"]
95             author = data["author"]
96 
97             title = textwrap.shorten(data["title"], width=64, placeholder="...")
98             link = self.URL + data["permalink"]
99 
100             embed.description += (
101                 f"[**{title}**]({link})\n"
102                 f"{text}"
103                 f"| {ups} upvotes | {comments} comments | u/{author} | {subreddit} |\n\n"
104             )
105 
106         embed.colour = Colour.blurple()
107 
108         return await channel.send(
109             content=content,
110             embed=embed
111         )
112 
113     async def poll_new_posts(self) -> None:
114         """Periodically search for new subreddit posts."""
115         while True:
116             await asyncio.sleep(RedditConfig.request_delay)
117 
118             for subreddit in RedditConfig.subreddits:
119                 # Make a HEAD request to the subreddit
120                 head_response = await self.bot.http_session.head(
121                     url=f"{self.URL}/{subreddit}/new.rss",
122                     headers=self.HEADERS
123                 )
124 
125                 content_length = head_response.headers["content-length"]
126 
127                 # If the content is the same size as before, assume there's no new posts.
128                 if content_length == self.prev_lengths.get(subreddit, None):
129                     continue
130 
131                 self.prev_lengths[subreddit] = content_length
132 
133                 # Now we can actually fetch the new data
134                 posts = await self.fetch_posts(f"{subreddit}/new")
135                 new_posts = []
136 
137                 # Only show new posts if we've checked before.
138                 if subreddit in self.last_ids:
139                     for post in posts:
140                         data = post["data"]
141 
142                         # Convert the ID to an integer for easy comparison.
143                         int_id = int(data["id"], 36)
144 
145                         # If we've already seen this post, finish checking
146                         if int_id <= self.last_ids[subreddit]:
147                             break
148 
149                         embed_data = {
150                             "title": textwrap.shorten(data["title"], width=64, placeholder="..."),
151                             "text": textwrap.shorten(data["selftext"], width=128, placeholder="..."),
152                             "url": self.URL + data["permalink"],
153                             "author": data["author"]
154                         }
155 
156                         new_posts.append(embed_data)
157 
158                 self.last_ids[subreddit] = int(posts[0]["data"]["id"], 36)
159 
160                 # Send all of the new posts as spicy embeds
161                 for data in new_posts:
162                     embed = Embed()
163 
164                     embed.title = data["title"]
165                     embed.url = data["url"]
166                     embed.description = data["text"]
167                     embed.set_footer(text=f"Posted by u/{data['author']} in {subreddit}")
168                     embed.colour = Colour.blurple()
169 
170                     await self.reddit_channel.send(embed=embed)
171 
172                 log.trace(f"Sent {len(new_posts)} new {subreddit} posts to channel {self.reddit_channel.id}.")
173 
174     async def poll_top_weekly_posts(self) -> None:
175         """Post a summary of the top posts every week."""
176         while True:
177             now = datetime.utcnow()
178 
179             # Calculate the amount of seconds until midnight next monday.
180             monday = now + timedelta(days=7 - now.weekday())
181             monday = monday.replace(hour=0, minute=0, second=0)
182             until_monday = (monday - now).total_seconds()
183 
184             await asyncio.sleep(until_monday)
185 
186             for subreddit in RedditConfig.subreddits:
187                 # Send and pin the new weekly posts.
188                 message = await self.send_top_posts(
189                     channel=self.reddit_channel,
190                     subreddit=subreddit,
191                     content=f"This week's top {subreddit} posts have arrived!",
192                     time="week"
193                 )
194 
195                 if subreddit.lower() == "r/python":
196                     # Remove the oldest pins so that only 5 remain at most.
197                     pins = await self.reddit_channel.pins()
198 
199                     while len(pins) >= 5:
200                         await pins[-1].unpin()
201                         del pins[-1]
202 
203                     await message.pin()
204 
205     @group(name="reddit", invoke_without_command=True)
206     async def reddit_group(self, ctx: Context) -> None:
207         """View the top posts from various subreddits."""
208         await ctx.invoke(self.bot.get_command("help"), "reddit")
209 
210     @reddit_group.command(name="top")
211     async def top_command(self, ctx: Context, subreddit: Subreddit = "r/Python") -> None:
212         """Send the top posts of all time from a given subreddit."""
213         await self.send_top_posts(
214             channel=ctx.channel,
215             subreddit=subreddit,
216             content=f"Here are the top {subreddit} posts of all time!",
217             time="all"
218         )
219 
220     @reddit_group.command(name="daily")
221     async def daily_command(self, ctx: Context, subreddit: Subreddit = "r/Python") -> None:
222         """Send the top posts of today from a given subreddit."""
223         await self.send_top_posts(
224             channel=ctx.channel,
225             subreddit=subreddit,
226             content=f"Here are today's top {subreddit} posts!",
227             time="day"
228         )
229 
230     @reddit_group.command(name="weekly")
231     async def weekly_command(self, ctx: Context, subreddit: Subreddit = "r/Python") -> None:
232         """Send the top posts of this week from a given subreddit."""
233         await self.send_top_posts(
234             channel=ctx.channel,
235             subreddit=subreddit,
236             content=f"Here are this week's top {subreddit} posts!",
237             time="week"
238         )
239 
240     @with_role(*STAFF_ROLES)
241     @reddit_group.command(name="subreddits", aliases=("subs",))
242     async def subreddits_command(self, ctx: Context) -> None:
243         """Send a paginated embed of all the subreddits we're relaying."""
244         embed = Embed()
245         embed.title = "Relayed subreddits."
246         embed.colour = Colour.blurple()
247 
248         await LinePaginator.paginate(
249             RedditConfig.subreddits,
250             ctx, embed,
251             footer_text="Use the reddit commands along with these to view their posts.",
252             empty=False,
253             max_lines=15
254         )
255 
256     @Cog.listener()
257     async def on_ready(self) -> None:
258         """Initiate reddit post event loop."""
259         self.reddit_channel = await self.bot.fetch_channel(Channels.reddit)
260 
261         if self.reddit_channel is not None:
262             if self.new_posts_task is None:
263                 self.new_posts_task = self.bot.loop.create_task(self.poll_new_posts())
264             if self.top_weekly_posts_task is None:
265                 self.top_weekly_posts_task = self.bot.loop.create_task(self.poll_top_weekly_posts())
266         else:
267             log.warning("Couldn't locate a channel for subreddit relaying.")
268 
269 
270 def setup(bot: Bot) -> None:
271     """Reddit cog load."""
272     bot.add_cog(Reddit(bot))
273     log.info("Cog loaded: Reddit")
274
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/bot/cogs/reddit.py b/bot/cogs/reddit.py
--- a/bot/cogs/reddit.py
+++ b/bot/cogs/reddit.py
@@ -21,6 +21,7 @@
 
     HEADERS = {"User-Agent": "Discord Bot: PythonDiscord (https://pythondiscord.com/)"}
     URL = "https://www.reddit.com"
+    MAX_FETCH_RETRIES = 3
 
     def __init__(self, bot: Bot):
         self.bot = bot
@@ -42,16 +43,23 @@
         if params is None:
             params = {}
 
-        response = await self.bot.http_session.get(
-            url=f"{self.URL}/{route}.json",
-            headers=self.HEADERS,
-            params=params
-        )
+        url = f"{self.URL}/{route}.json"
+        for _ in range(self.MAX_FETCH_RETRIES):
+            response = await self.bot.http_session.get(
+                url=url,
+                headers=self.HEADERS,
+                params=params
+            )
+            if response.status == 200 and response.content_type == 'application/json':
+                # Got appropriate response - process and return.
+                content = await response.json()
+                posts = content["data"]["children"]
+                return posts[:amount]
 
-        content = await response.json()
-        posts = content["data"]["children"]
+            await asyncio.sleep(3)
 
-        return posts[:amount]
+        log.debug(f"Invalid response from: {url} - status code {response.status}, mimetype {response.content_type}")
+        return list()  # Failed to get appropriate response within allowed number of retries.
 
     async def send_top_posts(
         self, channel: TextChannel, subreddit: Subreddit, content: str = None, time: str = "all"
@@ -62,13 +70,14 @@
         embed.description = ""
 
         # Get the posts
-        posts = await self.fetch_posts(
-            route=f"{subreddit}/top",
-            amount=5,
-            params={
-                "t": time
-            }
-        )
+        async with channel.typing():
+            posts = await self.fetch_posts(
+                route=f"{subreddit}/top",
+                amount=5,
+                params={
+                    "t": time
+                }
+            )
 
         if not posts:
             embed.title = random.choice(ERROR_REPLIES)

{"golden_diff": "diff --git a/bot/cogs/reddit.py b/bot/cogs/reddit.py\n--- a/bot/cogs/reddit.py\n+++ b/bot/cogs/reddit.py\n@@ -21,6 +21,7 @@\n \n     HEADERS = {\"User-Agent\": \"Discord Bot: PythonDiscord (https://pythondiscord.com/)\"}\n     URL = \"https://www.reddit.com\"\n+    MAX_FETCH_RETRIES = 3\n \n     def __init__(self, bot: Bot):\n         self.bot = bot\n@@ -42,16 +43,23 @@\n         if params is None:\n             params = {}\n \n-        response = await self.bot.http_session.get(\n-            url=f\"{self.URL}/{route}.json\",\n-            headers=self.HEADERS,\n-            params=params\n-        )\n+        url = f\"{self.URL}/{route}.json\"\n+        for _ in range(self.MAX_FETCH_RETRIES):\n+            response = await self.bot.http_session.get(\n+                url=url,\n+                headers=self.HEADERS,\n+                params=params\n+            )\n+            if response.status == 200 and response.content_type == 'application/json':\n+                # Got appropriate response - process and return.\n+                content = await response.json()\n+                posts = content[\"data\"][\"children\"]\n+                return posts[:amount]\n \n-        content = await response.json()\n-        posts = content[\"data\"][\"children\"]\n+            await asyncio.sleep(3)\n \n-        return posts[:amount]\n+        log.debug(f\"Invalid response from: {url} - status code {response.status}, mimetype {response.content_type}\")\n+        return list()  # Failed to get appropriate response within allowed number of retries.\n \n     async def send_top_posts(\n         self, channel: TextChannel, subreddit: Subreddit, content: str = None, time: str = \"all\"\n@@ -62,13 +70,14 @@\n         embed.description = \"\"\n \n         # Get the posts\n-        posts = await self.fetch_posts(\n-            route=f\"{subreddit}/top\",\n-            amount=5,\n-            params={\n-                \"t\": time\n-            }\n-        )\n+        async with channel.typing():\n+            posts = await self.fetch_posts(\n+                route=f\"{subreddit}/top\",\n+                amount=5,\n+                params={\n+                    \"t\": time\n+                }\n+            )\n \n         if not posts:\n             embed.title = random.choice(ERROR_REPLIES)\n", "issue": "The Reddit cog occassionally fails to relay new /r/python messages\nOccasionally, the [Reddit cog](/python-discord/bot/blob/master/bot/cogs/reddit.py) stops relaying new messages to the #reddit channel. From what I've seen, there are two ways in which the cog breaks and both should be easy to fix.\r\n\r\n#### 1. The background task fails on unexpected http responses\r\nThe background task to fetch new posts, the `self.new_posts_task` created in the `on_ready` event listener, assumes that the response we get from the Reddit API is valid `json` in [this line](/python-discord/bot/blob/master/bot/cogs/reddit.py#L55). If, for some temporary reason, we don't get a `json` response, but a `text/html` response (404, probably), the task fails with the following exception:\r\n\r\n```py\r\nIn [160]: bot.get_cog(\"Reddit\").new_posts_task\r\nOut[160]: \r\n  File \"/bot/bot/cogs/eval.py\", line 167, in _eval\r\n    res = await func()\r\n  File \"<string>\", line 8, in func\r\n  File \"/bot/bot/cogs/eval.py\", line 167, in _eval\r\n    res = await func()\r\n  File \"<string>\", line 8, in func\r\n  File \"/bot/bot/cogs/reddit.py\", line 142, in poll_new_posts\r\n    posts = await self.fetch_posts(f\"{subreddit}/new\")\r\n  File \"/bot/bot/cogs/reddit.py\", line 55, in fetch_posts\r\n    content = await response.json()\r\n  File \"/bot/.venv/lib/python3.6/site-packages/aiohttp/client_reqrep.py\", line 938, in json\r\n    headers=self.headers)\r\naiohttp.client_exceptions.ContentTypeError: 0, message='Attempt to decode JSON with unexpected mimetype: text/html; charset=utf-8'\r\n```\r\n\r\nThis line is in the `fetch_posts` utility function used by both the `new_posts_task` as well as the `top_weekly_posts_task`. This means that can break because of this error. \r\n\r\nThe likely solution is to either handle the exception or check the response status code before trying to parse it as json. We probably want to either have some kind of retry logic in the handling of the non-200 response, since otherwise the weekly top posts task skips a week.\r\n\r\n#### ~~2. The channel can't be initialized during `on_ready`~~\r\n\r\n~~Similar to the issue we had with the watch channels, it can occur that the bot's internal channel cache isn't fully loaded when the `on_ready` event fires. This means that the channel will not be retrieved and the tasks will never be started. This only happens intermittently, but it does occasionally happen.~~\r\n\r\n~~In such cases, the `self.bot.get_channel` returns `None`, because the channel has not been loaded into the internal bot cache yet at that point.~~\r\n\r\n```py\r\n    async def on_ready(self):\r\n        self.reddit_channel = self.bot.get_channel(Channels.reddit)\r\n```\r\n\r\n~~While this could be fixed by adding a slight delay or a bit of retry logic, another option is to wait for the migration to a later version of `discord.py` and use `await self.bot.fetch_channel` here instead. That will fetch the channel from the API, bypassing any issues we may have with the internal bot channel cache.~~\n", "before_files": [{"content": "import asyncio\nimport logging\nimport random\nimport textwrap\nfrom datetime import datetime, timedelta\nfrom typing import List\n\nfrom discord import Colour, Embed, Message, TextChannel\nfrom discord.ext.commands import Bot, Cog, Context, group\n\nfrom bot.constants import Channels, ERROR_REPLIES, Reddit as RedditConfig, STAFF_ROLES\nfrom bot.converters import Subreddit\nfrom bot.decorators import with_role\nfrom bot.pagination import LinePaginator\n\nlog = logging.getLogger(__name__)\n\n\nclass Reddit(Cog):\n    \"\"\"Track subreddit posts and show detailed statistics about them.\"\"\"\n\n    HEADERS = {\"User-Agent\": \"Discord Bot: PythonDiscord (https://pythondiscord.com/)\"}\n    URL = \"https://www.reddit.com\"\n\n    def __init__(self, bot: Bot):\n        self.bot = bot\n\n        self.reddit_channel = None\n\n        self.prev_lengths = {}\n        self.last_ids = {}\n\n        self.new_posts_task = None\n        self.top_weekly_posts_task = None\n\n    async def fetch_posts(self, route: str, *, amount: int = 25, params: dict = None) -> List[dict]:\n        \"\"\"A helper method to fetch a certain amount of Reddit posts at a given route.\"\"\"\n        # Reddit's JSON responses only provide 25 posts at most.\n        if not 25 >= amount > 0:\n            raise ValueError(\"Invalid amount of subreddit posts requested.\")\n\n        if params is None:\n            params = {}\n\n        response = await self.bot.http_session.get(\n            url=f\"{self.URL}/{route}.json\",\n            headers=self.HEADERS,\n            params=params\n        )\n\n        content = await response.json()\n        posts = content[\"data\"][\"children\"]\n\n        return posts[:amount]\n\n    async def send_top_posts(\n        self, channel: TextChannel, subreddit: Subreddit, content: str = None, time: str = \"all\"\n    ) -> Message:\n        \"\"\"Create an embed for the top posts, then send it in a given TextChannel.\"\"\"\n        # Create the new spicy embed.\n        embed = Embed()\n        embed.description = \"\"\n\n        # Get the posts\n        posts = await self.fetch_posts(\n            route=f\"{subreddit}/top\",\n            amount=5,\n            params={\n                \"t\": time\n            }\n        )\n\n        if not posts:\n            embed.title = random.choice(ERROR_REPLIES)\n            embed.colour = Colour.red()\n            embed.description = (\n                \"Sorry! We couldn't find any posts from that subreddit. \"\n                \"If this problem persists, please let us know.\"\n            )\n\n            return await channel.send(\n                embed=embed\n            )\n\n        for post in posts:\n            data = post[\"data\"]\n\n            text = data[\"selftext\"]\n            if text:\n                text = textwrap.shorten(text, width=128, placeholder=\"...\")\n                text += \"\\n\"  # Add newline to separate embed info\n\n            ups = data[\"ups\"]\n            comments = data[\"num_comments\"]\n            author = data[\"author\"]\n\n            title = textwrap.shorten(data[\"title\"], width=64, placeholder=\"...\")\n            link = self.URL + data[\"permalink\"]\n\n            embed.description += (\n                f\"[**{title}**]({link})\\n\"\n                f\"{text}\"\n                f\"| {ups} upvotes | {comments} comments | u/{author} | {subreddit} |\\n\\n\"\n            )\n\n        embed.colour = Colour.blurple()\n\n        return await channel.send(\n            content=content,\n            embed=embed\n        )\n\n    async def poll_new_posts(self) -> None:\n        \"\"\"Periodically search for new subreddit posts.\"\"\"\n        while True:\n            await asyncio.sleep(RedditConfig.request_delay)\n\n            for subreddit in RedditConfig.subreddits:\n                # Make a HEAD request to the subreddit\n                head_response = await self.bot.http_session.head(\n                    url=f\"{self.URL}/{subreddit}/new.rss\",\n                    headers=self.HEADERS\n                )\n\n                content_length = head_response.headers[\"content-length\"]\n\n                # If the content is the same size as before, assume there's no new posts.\n                if content_length == self.prev_lengths.get(subreddit, None):\n                    continue\n\n                self.prev_lengths[subreddit] = content_length\n\n                # Now we can actually fetch the new data\n                posts = await self.fetch_posts(f\"{subreddit}/new\")\n                new_posts = []\n\n                # Only show new posts if we've checked before.\n                if subreddit in self.last_ids:\n                    for post in posts:\n                        data = post[\"data\"]\n\n                        # Convert the ID to an integer for easy comparison.\n                        int_id = int(data[\"id\"], 36)\n\n                        # If we've already seen this post, finish checking\n                        if int_id <= self.last_ids[subreddit]:\n                            break\n\n                        embed_data = {\n                            \"title\": textwrap.shorten(data[\"title\"], width=64, placeholder=\"...\"),\n                            \"text\": textwrap.shorten(data[\"selftext\"], width=128, placeholder=\"...\"),\n                            \"url\": self.URL + data[\"permalink\"],\n                            \"author\": data[\"author\"]\n                        }\n\n                        new_posts.append(embed_data)\n\n                self.last_ids[subreddit] = int(posts[0][\"data\"][\"id\"], 36)\n\n                # Send all of the new posts as spicy embeds\n                for data in new_posts:\n                    embed = Embed()\n\n                    embed.title = data[\"title\"]\n                    embed.url = data[\"url\"]\n                    embed.description = data[\"text\"]\n                    embed.set_footer(text=f\"Posted by u/{data['author']} in {subreddit}\")\n                    embed.colour = Colour.blurple()\n\n                    await self.reddit_channel.send(embed=embed)\n\n                log.trace(f\"Sent {len(new_posts)} new {subreddit} posts to channel {self.reddit_channel.id}.\")\n\n    async def poll_top_weekly_posts(self) -> None:\n        \"\"\"Post a summary of the top posts every week.\"\"\"\n        while True:\n            now = datetime.utcnow()\n\n            # Calculate the amount of seconds until midnight next monday.\n            monday = now + timedelta(days=7 - now.weekday())\n            monday = monday.replace(hour=0, minute=0, second=0)\n            until_monday = (monday - now).total_seconds()\n\n            await asyncio.sleep(until_monday)\n\n            for subreddit in RedditConfig.subreddits:\n                # Send and pin the new weekly posts.\n                message = await self.send_top_posts(\n                    channel=self.reddit_channel,\n                    subreddit=subreddit,\n                    content=f\"This week's top {subreddit} posts have arrived!\",\n                    time=\"week\"\n                )\n\n                if subreddit.lower() == \"r/python\":\n                    # Remove the oldest pins so that only 5 remain at most.\n                    pins = await self.reddit_channel.pins()\n\n                    while len(pins) >= 5:\n                        await pins[-1].unpin()\n                        del pins[-1]\n\n                    await message.pin()\n\n    @group(name=\"reddit\", invoke_without_command=True)\n    async def reddit_group(self, ctx: Context) -> None:\n        \"\"\"View the top posts from various subreddits.\"\"\"\n        await ctx.invoke(self.bot.get_command(\"help\"), \"reddit\")\n\n    @reddit_group.command(name=\"top\")\n    async def top_command(self, ctx: Context, subreddit: Subreddit = \"r/Python\") -> None:\n        \"\"\"Send the top posts of all time from a given subreddit.\"\"\"\n        await self.send_top_posts(\n            channel=ctx.channel,\n            subreddit=subreddit,\n            content=f\"Here are the top {subreddit} posts of all time!\",\n            time=\"all\"\n        )\n\n    @reddit_group.command(name=\"daily\")\n    async def daily_command(self, ctx: Context, subreddit: Subreddit = \"r/Python\") -> None:\n        \"\"\"Send the top posts of today from a given subreddit.\"\"\"\n        await self.send_top_posts(\n            channel=ctx.channel,\n            subreddit=subreddit,\n            content=f\"Here are today's top {subreddit} posts!\",\n            time=\"day\"\n        )\n\n    @reddit_group.command(name=\"weekly\")\n    async def weekly_command(self, ctx: Context, subreddit: Subreddit = \"r/Python\") -> None:\n        \"\"\"Send the top posts of this week from a given subreddit.\"\"\"\n        await self.send_top_posts(\n            channel=ctx.channel,\n            subreddit=subreddit,\n            content=f\"Here are this week's top {subreddit} posts!\",\n            time=\"week\"\n        )\n\n    @with_role(*STAFF_ROLES)\n    @reddit_group.command(name=\"subreddits\", aliases=(\"subs\",))\n    async def subreddits_command(self, ctx: Context) -> None:\n        \"\"\"Send a paginated embed of all the subreddits we're relaying.\"\"\"\n        embed = Embed()\n        embed.title = \"Relayed subreddits.\"\n        embed.colour = Colour.blurple()\n\n        await LinePaginator.paginate(\n            RedditConfig.subreddits,\n            ctx, embed,\n            footer_text=\"Use the reddit commands along with these to view their posts.\",\n            empty=False,\n            max_lines=15\n        )\n\n    @Cog.listener()\n    async def on_ready(self) -> None:\n        \"\"\"Initiate reddit post event loop.\"\"\"\n        self.reddit_channel = await self.bot.fetch_channel(Channels.reddit)\n\n        if self.reddit_channel is not None:\n            if self.new_posts_task is None:\n                self.new_posts_task = self.bot.loop.create_task(self.poll_new_posts())\n            if self.top_weekly_posts_task is None:\n                self.top_weekly_posts_task = self.bot.loop.create_task(self.poll_top_weekly_posts())\n        else:\n            log.warning(\"Couldn't locate a channel for subreddit relaying.\")\n\n\ndef setup(bot: Bot) -> None:\n    \"\"\"Reddit cog load.\"\"\"\n    bot.add_cog(Reddit(bot))\n    log.info(\"Cog loaded: Reddit\")\n", "path": "bot/cogs/reddit.py"}], "after_files": [{"content": "import asyncio\nimport logging\nimport random\nimport textwrap\nfrom datetime import datetime, timedelta\nfrom typing import List\n\nfrom discord import Colour, Embed, Message, TextChannel\nfrom discord.ext.commands import Bot, Cog, Context, group\n\nfrom bot.constants import Channels, ERROR_REPLIES, Reddit as RedditConfig, STAFF_ROLES\nfrom bot.converters import Subreddit\nfrom bot.decorators import with_role\nfrom bot.pagination import LinePaginator\n\nlog = logging.getLogger(__name__)\n\n\nclass Reddit(Cog):\n    \"\"\"Track subreddit posts and show detailed statistics about them.\"\"\"\n\n    HEADERS = {\"User-Agent\": \"Discord Bot: PythonDiscord (https://pythondiscord.com/)\"}\n    URL = \"https://www.reddit.com\"\n    MAX_FETCH_RETRIES = 3\n\n    def __init__(self, bot: Bot):\n        self.bot = bot\n\n        self.reddit_channel = None\n\n        self.prev_lengths = {}\n        self.last_ids = {}\n\n        self.new_posts_task = None\n        self.top_weekly_posts_task = None\n\n    async def fetch_posts(self, route: str, *, amount: int = 25, params: dict = None) -> List[dict]:\n        \"\"\"A helper method to fetch a certain amount of Reddit posts at a given route.\"\"\"\n        # Reddit's JSON responses only provide 25 posts at most.\n        if not 25 >= amount > 0:\n            raise ValueError(\"Invalid amount of subreddit posts requested.\")\n\n        if params is None:\n            params = {}\n\n        url = f\"{self.URL}/{route}.json\"\n        for _ in range(self.MAX_FETCH_RETRIES):\n            response = await self.bot.http_session.get(\n                url=url,\n                headers=self.HEADERS,\n                params=params\n            )\n            if response.status == 200 and response.content_type == 'application/json':\n                # Got appropriate response - process and return.\n                content = await response.json()\n                posts = content[\"data\"][\"children\"]\n                return posts[:amount]\n\n            await asyncio.sleep(3)\n\n        log.debug(f\"Invalid response from: {url} - status code {response.status}, mimetype {response.content_type}\")\n        return list()  # Failed to get appropriate response within allowed number of retries.\n\n    async def send_top_posts(\n        self, channel: TextChannel, subreddit: Subreddit, content: str = None, time: str = \"all\"\n    ) -> Message:\n        \"\"\"Create an embed for the top posts, then send it in a given TextChannel.\"\"\"\n        # Create the new spicy embed.\n        embed = Embed()\n        embed.description = \"\"\n\n        # Get the posts\n        async with channel.typing():\n            posts = await self.fetch_posts(\n                route=f\"{subreddit}/top\",\n                amount=5,\n                params={\n                    \"t\": time\n                }\n            )\n\n        if not posts:\n            embed.title = random.choice(ERROR_REPLIES)\n            embed.colour = Colour.red()\n            embed.description = (\n                \"Sorry! We couldn't find any posts from that subreddit. \"\n                \"If this problem persists, please let us know.\"\n            )\n\n            return await channel.send(\n                embed=embed\n            )\n\n        for post in posts:\n            data = post[\"data\"]\n\n            text = data[\"selftext\"]\n            if text:\n                text = textwrap.shorten(text, width=128, placeholder=\"...\")\n                text += \"\\n\"  # Add newline to separate embed info\n\n            ups = data[\"ups\"]\n            comments = data[\"num_comments\"]\n            author = data[\"author\"]\n\n            title = textwrap.shorten(data[\"title\"], width=64, placeholder=\"...\")\n            link = self.URL + data[\"permalink\"]\n\n            embed.description += (\n                f\"[**{title}**]({link})\\n\"\n                f\"{text}\"\n                f\"| {ups} upvotes | {comments} comments | u/{author} | {subreddit} |\\n\\n\"\n            )\n\n        embed.colour = Colour.blurple()\n\n        return await channel.send(\n            content=content,\n            embed=embed\n        )\n\n    async def poll_new_posts(self) -> None:\n        \"\"\"Periodically search for new subreddit posts.\"\"\"\n        while True:\n            await asyncio.sleep(RedditConfig.request_delay)\n\n            for subreddit in RedditConfig.subreddits:\n                # Make a HEAD request to the subreddit\n                head_response = await self.bot.http_session.head(\n                    url=f\"{self.URL}/{subreddit}/new.rss\",\n                    headers=self.HEADERS\n                )\n\n                content_length = head_response.headers[\"content-length\"]\n\n                # If the content is the same size as before, assume there's no new posts.\n                if content_length == self.prev_lengths.get(subreddit, None):\n                    continue\n\n                self.prev_lengths[subreddit] = content_length\n\n                # Now we can actually fetch the new data\n                posts = await self.fetch_posts(f\"{subreddit}/new\")\n                new_posts = []\n\n                # Only show new posts if we've checked before.\n                if subreddit in self.last_ids:\n                    for post in posts:\n                        data = post[\"data\"]\n\n                        # Convert the ID to an integer for easy comparison.\n                        int_id = int(data[\"id\"], 36)\n\n                        # If we've already seen this post, finish checking\n                        if int_id <= self.last_ids[subreddit]:\n                            break\n\n                        embed_data = {\n                            \"title\": textwrap.shorten(data[\"title\"], width=64, placeholder=\"...\"),\n                            \"text\": textwrap.shorten(data[\"selftext\"], width=128, placeholder=\"...\"),\n                            \"url\": self.URL + data[\"permalink\"],\n                            \"author\": data[\"author\"]\n                        }\n\n                        new_posts.append(embed_data)\n\n                self.last_ids[subreddit] = int(posts[0][\"data\"][\"id\"], 36)\n\n                # Send all of the new posts as spicy embeds\n                for data in new_posts:\n                    embed = Embed()\n\n                    embed.title = data[\"title\"]\n                    embed.url = data[\"url\"]\n                    embed.description = data[\"text\"]\n                    embed.set_footer(text=f\"Posted by u/{data['author']} in {subreddit}\")\n                    embed.colour = Colour.blurple()\n\n                    await self.reddit_channel.send(embed=embed)\n\n                log.trace(f\"Sent {len(new_posts)} new {subreddit} posts to channel {self.reddit_channel.id}.\")\n\n    async def poll_top_weekly_posts(self) -> None:\n        \"\"\"Post a summary of the top posts every week.\"\"\"\n        while True:\n            now = datetime.utcnow()\n\n            # Calculate the amount of seconds until midnight next monday.\n            monday = now + timedelta(days=7 - now.weekday())\n            monday = monday.replace(hour=0, minute=0, second=0)\n            until_monday = (monday - now).total_seconds()\n\n            await asyncio.sleep(until_monday)\n\n            for subreddit in RedditConfig.subreddits:\n                # Send and pin the new weekly posts.\n                message = await self.send_top_posts(\n                    channel=self.reddit_channel,\n                    subreddit=subreddit,\n                    content=f\"This week's top {subreddit} posts have arrived!\",\n                    time=\"week\"\n                )\n\n                if subreddit.lower() == \"r/python\":\n                    # Remove the oldest pins so that only 5 remain at most.\n                    pins = await self.reddit_channel.pins()\n\n                    while len(pins) >= 5:\n                        await pins[-1].unpin()\n                        del pins[-1]\n\n                    await message.pin()\n\n    @group(name=\"reddit\", invoke_without_command=True)\n    async def reddit_group(self, ctx: Context) -> None:\n        \"\"\"View the top posts from various subreddits.\"\"\"\n        await ctx.invoke(self.bot.get_command(\"help\"), \"reddit\")\n\n    @reddit_group.command(name=\"top\")\n    async def top_command(self, ctx: Context, subreddit: Subreddit = \"r/Python\") -> None:\n        \"\"\"Send the top posts of all time from a given subreddit.\"\"\"\n        await self.send_top_posts(\n            channel=ctx.channel,\n            subreddit=subreddit,\n            content=f\"Here are the top {subreddit} posts of all time!\",\n            time=\"all\"\n        )\n\n    @reddit_group.command(name=\"daily\")\n    async def daily_command(self, ctx: Context, subreddit: Subreddit = \"r/Python\") -> None:\n        \"\"\"Send the top posts of today from a given subreddit.\"\"\"\n        await self.send_top_posts(\n            channel=ctx.channel,\n            subreddit=subreddit,\n            content=f\"Here are today's top {subreddit} posts!\",\n            time=\"day\"\n        )\n\n    @reddit_group.command(name=\"weekly\")\n    async def weekly_command(self, ctx: Context, subreddit: Subreddit = \"r/Python\") -> None:\n        \"\"\"Send the top posts of this week from a given subreddit.\"\"\"\n        await self.send_top_posts(\n            channel=ctx.channel,\n            subreddit=subreddit,\n            content=f\"Here are this week's top {subreddit} posts!\",\n            time=\"week\"\n        )\n\n    @with_role(*STAFF_ROLES)\n    @reddit_group.command(name=\"subreddits\", aliases=(\"subs\",))\n    async def subreddits_command(self, ctx: Context) -> None:\n        \"\"\"Send a paginated embed of all the subreddits we're relaying.\"\"\"\n        embed = Embed()\n        embed.title = \"Relayed subreddits.\"\n        embed.colour = Colour.blurple()\n\n        await LinePaginator.paginate(\n            RedditConfig.subreddits,\n            ctx, embed,\n            footer_text=\"Use the reddit commands along with these to view their posts.\",\n            empty=False,\n            max_lines=15\n        )\n\n    @Cog.listener()\n    async def on_ready(self) -> None:\n        \"\"\"Initiate reddit post event loop.\"\"\"\n        self.reddit_channel = await self.bot.fetch_channel(Channels.reddit)\n\n        if self.reddit_channel is not None:\n            if self.new_posts_task is None:\n                self.new_posts_task = self.bot.loop.create_task(self.poll_new_posts())\n            if self.top_weekly_posts_task is None:\n                self.top_weekly_posts_task = self.bot.loop.create_task(self.poll_top_weekly_posts())\n        else:\n            log.warning(\"Couldn't locate a channel for subreddit relaying.\")\n\n\ndef setup(bot: Bot) -> None:\n    \"\"\"Reddit cog load.\"\"\"\n    bot.add_cog(Reddit(bot))\n    log.info(\"Cog loaded: Reddit\")\n", "path": "bot/cogs/reddit.py"}]}