DEVELOPMENT.md

Development

Design Decisions

We specifically opt for a single-space leaderboard for simplicity. We solve the issue of keeping the gradio UI interactive while models are evaluating by using multiprocessing instead of a separate space. Leaderboard entries are persisted in a Huggingface Dataset to avoid paying for persistent storage. Tasks are deliberately ephemeral.

Local Setup

Prerequisites

• Python 3.10
• Git
• A love for speech recognition! 🎤

Quick Installation

0. Make sure git-lfs is installed (https://git-lfs.com)

git lfs install

1. Clone this repository:

git clone https://huggingface.co/spaces/KoelLabs/IPA-Transcription-EN

2. Setup your environment:

# Create a virtual environment with Python 3.10
python3.10 -m venv venv

# Activate the virtual environment
. ./venv/bin/activate
# use `deactivate` to exit out of it

# Install the required dependencies
pip install -r requirements_lock.txt

# Add a HF_TOKEN with access to your backing dataset (in app/hf.py) and any models you want to be able to run
huggingface-cli login

3. Launch the leaderboard:

. ./scripts/run-dev.sh      # development mode (auto-reloads)
. ./scripts/run-prod.sh     # production mode (no auto-reloads)

4. Visit http://localhost:7860 in your browser and see the magic! ✨

Adding New Datasets

The datasets are pre-processed into a single dataset stored in app/data/test with three columns: audio (16 kHz), ipa, and dataset (original source). This is done using the scripts/sample_test_set.py file. To add new datasets, add them to this script. Beware that existing leaderboard entries will need to be recalculated. You can do this locally by accessing the dataset corresponding to LEADERBOARD_ID stored in app/hf.py.

Adding/Removing Dependencies

0. Activate the virtual environment with . ./venv/bin/activate
1. Add the dependency to requirements.txt (or remove it)
2. Make sure you have no unused dependencies with pipx run deptry . (if necessary python -m pip install pipx)
3. Run pip install -r requirements.txt
4. Freeze the dependencies with pip freeze > requirements_lock.txt

Forking Into Your Own Leaderboard

0. Navigate to the space, click the three dots on the right and select Duplicate this Space
1. Modify the LEADERBOARD_ID in app/hf.py to be some dataset that you own that the new space can use to store data. You don't need to create the dataset but if you do, it should be empty.
2. Open the settings in your new space and add a new secret HF_TOKEN. You can create it here. It just needs read access to all models you want to add to the leaderboard and write access to the private backing dataset specified by LEADERBOARD_ID.
3. Submit some models and enjoy!

File Structure

The two most imporant files are app/app.py for the main gradio UI and app/tasks.py for the background tasks that evaluate models.

IPA-Transcription-EN/
├── README.md                   # General information about the leaderboard
├── CONTRIBUTING.md             # Contribution guidelines
├── DEVELOPMENT.md              # Development setup and design decisions
├── requirements.txt            # Python dependencies
├── requirements_lock.txt       # Locked dependencies
├── scripts                     # Helper scripts    
│   ├── sample_test_set.py      # Compute the combined test set
│   ├── run-prod.sh             # Run the leaderboard in production mode
│   └── run-dev.sh              # Run the leaderboard in development mode
├── venv                        # Virtual environment
├── app/                        # All application code lives here
│   ├── data/                   # Phoneme transcription test set
│   ├── app.py                  # Main Gradio UI
│   ├── hf.py                   # Interface with the Huggingface API
│   ├── inference.py            # Model inference
│   └── metrics.py              # Evaluation metrics
│   ├── tasks.py                # Background tasks for model evaluation
└── img/                        # Images for README and other documentation