Update custom model files, README, and requirements

README.md

@@ -1,75 +1,54 @@
 ---
-library_name: transformers
-tags:
-- generated_from_trainer
+license: mit
+language:
+- en
 datasets:
-- generator
-model-index:
-- name: tiny-audio-swiglu
-  results: []
+- speechbrain/LoquaciousSet
+base_model:
+- facebook/hubert-xlarge-ls960-ft
+- HuggingFaceTB/SmolLM3-3B
+pipeline_tag: automatic-speech-recognition
+tags:
+- asr
+- speech-recognition
+- audio
+- smollm
+- hubert
 ---
 
-<!-- This model card has been generated automatically according to the information the Trainer had access to. You
-should probably proofread and complete it, then remove this comment. -->
-
-# tiny-audio-swiglu
+# Tiny Audio Model Card
 
-This model is a fine-tuned version of [](https://huggingface.co/) on the generator dataset.
-It achieves the following results on the evaluation set:
-- Loss: 1.8530
+This model was born from a simple idea: what if anyone could train a powerful, modern speech recognition model for the price of a few coffees? This model is the result of the [Tiny Audio course](https://github.com/alexkroman/tiny-audio/blob/main/docs/course/0-course-overview.md), a free, hands-on guide to building your own ASR system from scratch.
 
-## Model description
+## The Story of this Model
 
-More information needed
+This model isn't the product of a massive research lab with an unlimited budget. It's the result of a 24-hour training run on a single GPU, made possible by an efficient projector-only training approach. By combining the strengths of a massive pretrained audio encoder (`facebook/hubert-xlarge-ls960-ft`) and a powerful language model (`HuggingFaceTB/SmolLM3-3B`), and only training a small projector between them, we can create a high-quality ASR model with minimal resources.
 
-## Intended uses & limitations
+This model is a testament to the power of open-source and the incredible tools and models that are now available to everyone.
 
-More information needed
+## Intended Use
 
-## Training and evaluation data
+This model is for you. It's for the curious, the builders, the learners. It's for anyone who wants to understand how modern AI works by getting their hands dirty. Use it to transcribe your podcasts, your meetings, your voice memos. But more importantly, use it as a starting point. Fork it, fine-tune it, break it, and make it your own.
 
-More information needed
+## Performance
 
-## Training procedure
+This model achieves a Word Error Rate (WER) of **12.14%** on the LoquaciousSet test set. It's not perfect, but it's a solid baseline that you can build on. See how it compares to other models on the [community leaderboard](https://github.com/alexkroman/tiny-audio#leaderboard).
 
-### Training hyperparameters
+## How to Use
 
-The following hyperparameters were used during training:
-- learning_rate: 0.0002
-- train_batch_size: 8
-- eval_batch_size: 24
-- seed: 43
-- gradient_accumulation_steps: 3
-- total_train_batch_size: 24
-- optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
-- lr_scheduler_type: cosine
-- lr_scheduler_warmup_steps: 1000
-- training_steps: 15000
+```python
+from transformers import pipeline
 
-### Training results
+pipe = pipeline("automatic-speech-recognition", model="mazesmazes/tiny-audio", trust_remote_code=True)
 
-| Training Loss | Epoch  | Step  | Validation Loss |
-|:-------------:|:------:|:-----:|:---------------:|
-| 6.1089        | 0.0667 | 1000  | 2.0578          |
-| 5.7958        | 0.1333 | 2000  | 1.9403          |
-| 5.7099        | 0.2    | 3000  | 1.9094          |
-| 5.6074        | 0.2667 | 4000  | 1.8903          |
-| 5.5041        | 0.3333 | 5000  | 1.8801          |
-| 5.7084        | 0.4    | 6000  | 1.8671          |
-| 5.6099        | 0.4667 | 7000  | 1.8626          |
-| 5.5155        | 0.5333 | 8000  | 1.8574          |
-| 5.551         | 0.6    | 9000  | 1.8548          |
-| 5.4795        | 0.6667 | 10000 | 1.8514          |
-| 5.5731        | 0.7333 | 11000 | 1.8507          |
-| 5.5508        | 0.8    | 12000 | 1.8504          |
-| 5.4282        | 0.8667 | 13000 | 1.8501          |
-| 5.5229        | 0.0667 | 14000 | 1.8529          |
-| 5.3341        | 0.1333 | 15000 | 1.8530          |
+result = pipe("path/to/audio.wav")
+print(result["text"])
+```
 
+## How to Get Involved
 
-### Framework versions
+This project is more than just a model; it's a community. Here's how you can get involved:
 
-- Transformers 4.57.1
-- Pytorch 2.8.0+cu128
-- Datasets 4.4.1
-- Tokenizers 0.22.1
+- **Take the course**: The best way to start is to go through the [free 6-hour course](https://github.com/alexkroman/tiny-audio/blob/main/docs/course/0-course-overview.md) and train your own model.
+- **Share your results**: Add your model to the [leaderboard](https://github.com/alexkroman/tiny-audio#leaderboard) and share what you've learned.
+- **Join the conversation**: Ask questions, share your ideas, and connect with other builders in the [GitHub Discussions](https://github.com/alexkroman/tiny-audio/discussions).

asr_config.py

@@ -45,10 +45,10 @@ class ASRConfig(transformers.PretrainedConfig):
         # Set default generation parameters
         generation_defaults = {
             "num_beams": 1,
-            "max_new_tokens": 128,
+            "max_new_tokens": 96,
             "min_new_tokens": 1,
             "do_sample": False,
-            "repetition_penalty": 1.0,
+            "repetition_penalty": 1.05,
             "length_penalty": 1.0,
             "no_repeat_ngram_size": 0,
             "use_cache": True,

handler.py

@@ -0,0 +1,132 @@
+"""Custom inference handler for HuggingFace Inference Endpoints."""
+
+from typing import Any, Dict, List, Union
+
+import torch
+
+try:
+    # For remote execution, imports are relative
+    from .asr_modeling import ASRModel
+    from .asr_pipeline import ASRPipeline
+except ImportError:
+    # For local execution, imports are not relative
+    from asr_modeling import ASRModel  # type: ignore[no-redef]
+    from asr_pipeline import ASRPipeline  # type: ignore[no-redef]
+
+
+class EndpointHandler:
+    def __init__(self, path: str = ""):
+        import os
+
+        import nltk
+
+        nltk.download("punkt_tab", quiet=True)
+
+        os.environ.setdefault("PYTORCH_CUDA_ALLOC_CONF", "expandable_segments:True")
+
+        # Enable TF32 for faster matmul on Ampere+ GPUs (A100, etc.)
+        # Also beneficial for T4 (Turing) which supports TensorFloat-32
+        torch.backends.cuda.matmul.allow_tf32 = True
+        torch.backends.cudnn.allow_tf32 = True
+
+        # Set device and dtype
+        self.device = "cuda" if torch.cuda.is_available() else "cpu"
+
+        # Use float16 for better T4 compatibility (bfloat16 not well supported on T4)
+        # T4 has excellent float16 performance with tensor cores
+        self.dtype = torch.float16 if self.device == "cuda" else torch.float32
+
+        # Enable CUDA optimizations
+        if torch.cuda.is_available():
+            torch.backends.cudnn.benchmark = True
+
+        # Prepare model kwargs for pipeline
+        model_kwargs = {
+            "dtype": self.dtype,
+            "low_cpu_mem_usage": True,
+        }
+        if torch.cuda.is_available():
+            model_kwargs["attn_implementation"] = (
+                "flash_attention_2" if self._is_flash_attn_available() else "sdpa"
+            )
+
+        # Load model (this loads the model, tokenizer, and feature extractor)
+        self.model = ASRModel.from_pretrained(path, **model_kwargs)
+
+        # Instantiate custom pipeline - it will get feature_extractor and tokenizer from model
+        self.pipe = ASRPipeline(
+            model=self.model,
+            feature_extractor=self.model.feature_extractor,
+            tokenizer=self.model.tokenizer,
+            device=self.device,
+        )
+
+        # Apply torch.compile if enabled (after model is loaded by pipeline)
+        # Use "default" mode for T4 - better compatibility than "reduce-overhead"
+        # "reduce-overhead" is better for A100+ but can be slower on older GPUs
+        if torch.cuda.is_available() and os.getenv("ENABLE_TORCH_COMPILE", "1") == "1":
+            compile_mode = os.getenv("TORCH_COMPILE_MODE", "default")
+            self.model = torch.compile(self.model, mode=compile_mode)
+            self.pipe.model = self.model
+
+        # Warmup the model to trigger compilation and optimize kernels
+        if torch.cuda.is_available():
+            self._warmup()
+
+    def _is_flash_attn_available(self):
+        """Check if flash attention is available."""
+        import importlib.util
+
+        return importlib.util.find_spec("flash_attn") is not None
+
+    def _warmup(self):
+        """Warmup to trigger model compilation and allocate GPU memory."""
+        try:
+            # Create dummy audio (1 second at config sample rate)
+            sample_rate = self.pipe.model.config.audio_sample_rate
+            dummy_audio = torch.randn(sample_rate, dtype=torch.float32)
+
+            # Run inference to trigger torch.compile and kernel optimization
+            with torch.inference_mode():
+                warmup_tokens = self.pipe.model.config.inference_warmup_tokens
+                _ = self.pipe(
+                    {"raw": dummy_audio, "sampling_rate": sample_rate},
+                    max_new_tokens=warmup_tokens,
+                )
+
+            # Force CUDA synchronization to ensure kernels are compiled
+            if torch.cuda.is_available():
+                torch.cuda.synchronize()
+                # Clear cache after warmup to free memory
+                torch.cuda.empty_cache()
+
+        except Exception as e:
+            print(f"Warmup skipped due to: {e}")
+
+    def __call__(self, data: Dict[str, Any]) -> Union[Dict[str, Any], List[Dict[str, Any]]]:
+        inputs = data.get("inputs")
+        if inputs is None:
+            raise ValueError("Missing 'inputs' in request data")
+
+        params = data.get("parameters", {})
+        max_new_tokens = params.get("max_new_tokens", 128)
+        num_beams = params.get("num_beams", 1)
+        do_sample = params.get("do_sample", False)
+        length_penalty = params.get("length_penalty", 1.0)
+        repetition_penalty = params.get("repetition_penalty", 1.05)
+        no_repeat_ngram_size = params.get("no_repeat_ngram_size", 0)
+        early_stopping = params.get("early_stopping", True)
+        default_diversity = self.pipe.model.config.inference_diversity_penalty
+        diversity_penalty = params.get("diversity_penalty", default_diversity)
+
+        return self.pipe(
+            inputs,
+            max_new_tokens=max_new_tokens,
+            num_beams=num_beams,
+            do_sample=do_sample,
+            length_penalty=length_penalty,
+            repetition_penalty=repetition_penalty,
+            no_repeat_ngram_size=no_repeat_ngram_size,
+            early_stopping=early_stopping,
+            diversity_penalty=diversity_penalty,
+        )

requirements.txt

@@ -0,0 +1,6 @@
+# Core dependencies for tiny-audio model inference
+# This file is pushed to HuggingFace for model repository
+
+# Transformers - main library for model loading and inference
+transformers>=4.57.0
+truecase