README.md

---
title: InferenceProviderTestingBackend
emoji: 📈
colorFrom: yellow
colorTo: indigo
sdk: gradio
sdk_version: 5.49.1
app_file: app.py
pinned: false
---

# Inference Provider Testing Dashboard

A Gradio-based dashboard for launching and monitoring evaluation jobs across multiple models and inference providers using Hugging Face's job API.

## Setup

### Prerequisites

- Python 3.8+
- Hugging Face account with API token
- Access to the `IPTesting` namespace on Hugging Face

### Installation

1. Clone or navigate to this repository:
```bash
cd InferenceProviderTestingBackend
```

2. Install dependencies:
```bash
pip install -r requirements.txt
```

3. Set up your Hugging Face token as an environment variable:
```bash
export HF_TOKEN="your_huggingface_token_here"
```

**Important**: Your HF_TOKEN must have:
- Permission to call inference providers
- Write access to the `IPTesting` organization

## Usage

### Starting the Dashboard

Run the Gradio app:
```bash
python app.py
```

### Initialize Models and Providers

1. Click the **"Fetch and Initialize Models/Providers"** button to automatically populate the `models_providers.txt` file with popular models and their available inference providers.

2. Alternatively, manually edit `models_providers.txt` with your desired model-provider combinations:
```
meta-llama/Llama-3.2-3B-Instruct  fireworks-ai
meta-llama/Llama-3.2-3B-Instruct  together-ai
Qwen/Qwen2.5-7B-Instruct  fireworks-ai
mistralai/Mistral-7B-Instruct-v0.3  together-ai
```

Format: `model_name  provider_name` (separated by spaces or tabs)

### Launching Jobs

1. Enter the evaluation tasks in the **Tasks** field (e.g., `lighteval|mmlu|0|0`)
2. Verify the config file path (default: `models_providers.txt`)
3. Click **"Launch Jobs"**

The system will:
- Read all model-provider combinations from the config file
- Launch a separate evaluation job for each combination
- Log the job ID and status
- Monitor job progress automatically

### Monitoring Jobs

The **Job Results** table displays all jobs with:
- **Model**: The model being tested
- **Provider**: The inference provider
- **Last Run**: Timestamp of when the job was last launched
- **Status**: Current status (running/complete/failed/cancelled)
- **Current Score**: Average score from the most recent run
- **Previous Score**: Average score from the prior run (for comparison)
- **Latest Job Id**: Latest job id to put in https://huggingface.co/jobs/NAMESPACE/JOBID for inspection

The table auto-refreshes every 30 seconds, or you can click "Refresh Results" for manual updates.

## Configuration

### Tasks Format

The tasks parameter follows the lighteval format. Examples:
- `lighteval|mmlu|0` - MMLU benchmark

### Daily Checkpoint

The system automatically saves all results to the HuggingFace dataset at **00:00 (midnight)** every day. 

### Data Persistence

All job results are stored in a HuggingFace dataset (`IPTesting/inference-provider-test-results`), which means:
- Results persist across app restarts
- Historical score comparisons are maintained
- Data can be accessed programmatically via the HF datasets library

## Architecture

- **Main Thread**: Runs the Gradio interface
- **Monitor Thread**: Updates job statuses every 30 seconds and extracts scores from completed jobs
- **APScheduler**: Background scheduler that handles daily checkpoint saves at midnight (cron-based)
- **Thread-safe**: Uses locks to prevent access issues when checking job_results
- **HuggingFace Dataset Storage**: Persists results to `IPTesting/inference-provider-test-results` dataset

## Troubleshooting

### Jobs Not Launching

- Verify your `HF_TOKEN` is set and has the required permissions
- Check that the `IPTesting` namespace exists and you have access
- Review logs for specific error messages

### Scores Not Appearing

- Scores are extracted from job logs after completion
- The extraction parses the results table that appears in job logs
- It extracts the score for each task (from the first row where the task name appears)
- The final score is the average of all task scores
- Example table format:
  ```
  | Task                    | Version | Metric                | Value  | Stderr |
  | extended:ifeval:0       |         | prompt_level_strict_acc | 0.9100 | 0.0288 |
  | lighteval:gpqa:diamond:0 |        | gpqa_pass@k_with_k     | 0.5000 | 0.0503 |
  ```
- If scores don't appear, check console output for extraction errors or parsing issues

## Files

- [app.py](app.py) - Main Gradio application with UI and job management
- [utils/](utils/) - Utility package with helper modules:
  - [utils/io.py](utils/io.py) - I/O operations: model/provider fetching, file operations, dataset persistence
  - [utils/jobs.py](utils/jobs.py) - Job management: launching, monitoring, score extraction
- [models_providers.txt](models_providers.txt) - Configuration file with model-provider combinations
- [requirements.txt](requirements.txt) - Python dependencies
- [README.md](README.md) - This file