Process webhooks in the background

(not tested but "should work")

When receiving a webhook, you should respond in less than 30s. If process takes longer than that, it's best to run it in a background task. You can use `fastapi.BackgroundTasks` for that. When you receive a webhook, you quickly check if it's a valid payload or not. If accepted, you schedule a task and return HTTP 202 (Accepted). Otherwise, you simply return a HTTP 200.

Inspired by https://huggingface.co/spaces/Wauplin/gradio-space-ci/blob/main/src/gradio_space_ci/webhook.py#L241.

app.py

@@ -1,6 +1,7 @@
 import os
 from pathlib import Path
 
+from fastapi import BackgroundTasks, Response, status
 import gradio as gr
 from huggingface_hub import WebhookPayload, WebhooksServer
 
@@ -72,12 +73,15 @@ app = WebhooksServer(ui=ui.queue(), webhook_secret=WEBHOOK_SECRET)
 
 
 @app.add_webhook("/dataset_repo")
-async def community(payload: WebhookPayload):
-    if payload.event.scope.startswith("repo"):
-        logger.info(f"Webhook received from {payload.repo.name} indicating a repo {payload.event.action}")
-    else:
-        return
+async def community(payload: WebhookPayload, task_queue: BackgroundTasks):
+    if not payload.event.scope.startswith("repo"):
+        return Response("No task scheduled", status_code=status.HTTP_200_OK)
 
+    logger.info(f"Webhook received from {payload.repo.name} indicating a repo {payload.event.action}")
+    task_queue.add_task(_process_webhook, payload=payload)
+    return Response("Task scheduled.", status_code=status.HTTP_202_ACCEPTED)
+
+def _process_webhook(payload: WebhookPayload):
     logger.info(f"Loading new dataset...")
     dataset, original_dataset = load_datasets()
     logger.info(f"Loaded new dataset")