LOGGING_GUIDE.md

Comprehensive Logging Guide

The Video Transcription Service now includes detailed step-by-step logging to help you monitor and debug transcription progress.

🎯 What You Can Track

Complete Transcription Journey

• ✅ File upload and validation
• ✅ Video processing steps
• ✅ Whisper model loading
• ✅ Audio extraction progress
• ✅ Transcription inference
• ✅ Results and cleanup
• ✅ Error handling and debugging

Real-time Progress Monitoring

• 📊 Processing times for each step
• 📏 File sizes and durations
• 🌐 Language detection
• 📝 Text length and previews
• ⚠️ Warnings and errors

🚀 Quick Start

Basic Logging (Default)

python main.py

Debug Mode (Detailed Logs)

DEBUG=true python main.py

Log to File

LOG_TO_FILE=true python main.py

Combined (Debug + File)

DEBUG=true LOG_TO_FILE=true python main.py

📊 Real-time Monitoring

Monitor Service Health

python log_monitor.py test

Upload and Monitor Video

python log_monitor.py upload video.mp4

Monitor Existing Transcription

python log_monitor.py monitor 123

📋 Sample Log Output

Service Startup

2024-01-15 10:30:00 - main - INFO - 🚀 Starting Video Transcription Service
2024-01-15 10:30:00 - main - INFO - ==================================================
2024-01-15 10:30:00 - main - INFO - 📋 Service Configuration:
2024-01-15 10:30:00 - main - INFO -    🤖 Whisper Model: base
2024-01-15 10:30:00 - main - INFO -    📏 Max File Size: 100MB
2024-01-15 10:30:00 - main - INFO -    🕒 Cleanup Interval: 3.5 hours
2024-01-15 10:30:00 - main - INFO -    🚦 Rate Limit: 10 requests/minute
2024-01-15 10:30:00 - main - INFO -    🌐 Host: 0.0.0.0:8000
2024-01-15 10:30:00 - main - INFO -    📁 Supported Formats: .mp4, .avi, .mov, .mkv, .wmv, .flv, .webm, .m4v
2024-01-15 10:30:00 - main - INFO - ==================================================

File Upload Process

2024-01-15 10:30:15 - main - INFO - 🚀 Starting transcription request for file: video.mp4
2024-01-15 10:30:15 - main - INFO - 🌐 Language specified: auto-detect
2024-01-15 10:30:15 - main - INFO - 📁 Validating file: video.mp4
2024-01-15 10:30:15 - main - INFO - 🔍 File extension: .mp4
2024-01-15 10:30:15 - main - INFO - ✅ File format validation passed: .mp4
2024-01-15 10:30:15 - main - INFO - 📊 Reading file content for size validation...
2024-01-15 10:30:15 - main - INFO - 📏 File size: 25.34MB (max: 100MB)
2024-01-15 10:30:15 - main - INFO - ✅ File size validation passed: 25.34MB

Storage Operations

2024-01-15 10:30:15 - storage - INFO - 📝 Creating new transcription entry with ID: 1
2024-01-15 10:30:15 - storage - INFO - 🌐 Language: auto-detect
2024-01-15 10:30:15 - storage - INFO - ✅ Transcription 1 created successfully
2024-01-15 10:30:15 - storage - INFO - 📊 Total active transcriptions: 1

Video Processing

2024-01-15 10:30:15 - transcription_service - INFO - 🎬 Starting video transcription for ID: 1
2024-01-15 10:30:15 - transcription_service - INFO - 📊 Video size: 25.34MB
2024-01-15 10:30:15 - transcription_service - INFO - 🌐 Language: auto-detect
2024-01-15 10:30:15 - transcription_service - INFO - 📝 Updating status to PROCESSING for ID: 1

Model Loading (First Time)

2024-01-15 10:30:15 - transcription_service - INFO - 🤖 Loading Whisper model: base
2024-01-15 10:30:15 - transcription_service - INFO - 📥 This may take 30-60 seconds for first-time download...
2024-01-15 10:30:45 - transcription_service - INFO - ✅ Whisper model loaded successfully in 30.2 seconds

Audio Extraction

2024-01-15 10:30:45 - transcription_service - INFO - 🎵 Extracting audio from video for transcription 1
2024-01-15 10:30:45 - transcription_service - INFO - 📁 Creating temporary video file...
2024-01-15 10:30:45 - transcription_service - INFO - 📁 Temporary files created - Video: /tmp/xyz.tmp, Audio: /tmp/abc.wav
2024-01-15 10:30:45 - transcription_service - INFO - 🎵 Running FFmpeg to extract audio...
2024-01-15 10:30:45 - transcription_service - INFO - 🔧 Configuring FFmpeg for audio extraction...
2024-01-15 10:30:45 - transcription_service - INFO -    - Codec: PCM 16-bit
2024-01-15 10:30:45 - transcription_service - INFO -    - Channels: 1 (mono)
2024-01-15 10:30:45 - transcription_service - INFO -    - Sample rate: 16kHz
2024-01-15 10:30:48 - transcription_service - INFO - ✅ FFmpeg audio extraction completed
2024-01-15 10:30:48 - transcription_service - INFO - ✅ Audio extraction successful - Size: 8.45MB
2024-01-15 10:30:48 - transcription_service - INFO - ✅ Audio extraction completed in 3.1 seconds

Transcription Process

2024-01-15 10:30:48 - transcription_service - INFO - 🗣️ Starting audio transcription for ID 1
2024-01-15 10:30:48 - transcription_service - INFO - 🗣️ Starting Whisper transcription...
2024-01-15 10:30:48 - transcription_service - INFO - 🎵 Audio file: /tmp/abc.wav
2024-01-15 10:30:48 - transcription_service - INFO - 🌐 Language: auto-detect
2024-01-15 10:30:48 - transcription_service - INFO - ⚡ Running transcription in background thread...
2024-01-15 10:30:48 - transcription_service - INFO - 🤖 Preparing Whisper transcription options...
2024-01-15 10:30:48 - transcription_service - INFO - 🌐 Language: auto-detect
2024-01-15 10:30:48 - transcription_service - INFO - 🎯 Starting Whisper model inference...
2024-01-15 10:31:15 - transcription_service - INFO - ✅ Whisper inference completed in 27.3 seconds
2024-01-15 10:31:15 - transcription_service - INFO - 📝 Text length: 1247 characters
2024-01-15 10:31:15 - transcription_service - INFO - 🌐 Detected language: en
2024-01-15 10:31:15 - transcription_service - INFO - ⏱️ Audio duration: 180.50 seconds
2024-01-15 10:31:15 - transcription_service - INFO - 📄 Text preview: Hello, welcome to this video tutorial where we'll be discussing...

Completion

2024-01-15 10:31:15 - transcription_service - INFO - ✅ Transcription completed in 27.3 seconds
2024-01-15 10:31:15 - transcription_service - INFO - 💾 Saving transcription results for ID 1
2024-01-15 10:31:15 - storage - INFO - 📝 Updated transcription 1
2024-01-15 10:31:15 - storage - INFO - 🔄 Status changed: processing → completed
2024-01-15 10:31:15 - storage - INFO - 📄 Text updated: Hello, welcome to this video tutorial where we'll...
2024-01-15 10:31:15 - transcription_service - INFO - 🧹 Cleaning up temporary audio file
2024-01-15 10:31:15 - transcription_service - INFO - 🎉 Transcription 1 completed successfully in 60.2 seconds total

🔧 Log Levels

INFO (Default)

• Service startup/shutdown
• Request processing
• Status updates
• Completion messages

DEBUG (Detailed)

• File validation details
• Temporary file paths
• FFmpeg configuration
• Model loading progress
• Memory usage info

WARNING

• Large file warnings
• Performance issues
• Non-critical errors

ERROR

• Processing failures
• File format issues
• System errors
• Transcription failures

📁 Log Files

When LOG_TO_FILE=true, logs are saved to:

transcription_service_YYYYMMDD_HHMMSS.log

Example: transcription_service_20240115_103000.log

🛠️ Troubleshooting with Logs

Common Issues and Log Patterns

1. NumPy Compatibility Error

ERROR - A module that was compiled using NumPy 1.x cannot be run in NumPy 2.2.6

Solution: Run python fix_numpy.py

2. FFmpeg Not Found

ERROR - FFmpeg audio extraction failed: [Errno 2] No such file or directory: 'ffmpeg'

Solution: Install FFmpeg for your OS

3. File Too Large

ERROR - File too large: 150.5MB > 100MB

Solution: Compress video or increase limit in config.py

4. Model Loading Issues

ERROR - Failed to load Whisper model: [Errno 28] No space left on device

Solution: Free up disk space or use smaller model

5. Memory Issues

ERROR - Process killed (signal 9)

Solution: Use smaller files or increase available memory

🎯 Performance Monitoring

Key Metrics to Watch

• Model Loading Time: Should be 15-60 seconds (first time only)
• Audio Extraction: Usually 1-5 seconds per minute of video
• Transcription Speed: Varies by model and content (typically 0.1-0.5x real-time)
• Memory Usage: Monitor for large files
• Active Transcriptions: Track concurrent processing

Optimization Tips

• Use tiny model for faster processing
• Compress videos before upload
• Monitor memory usage with large files
• Use DEBUG mode to identify bottlenecks

📊 Integration Examples

Parse Logs Programmatically

import re
from datetime import datetime

def parse_transcription_logs(log_file):
    with open(log_file, 'r') as f:
        for line in f:
            if 'Transcription' in line and 'completed successfully' in line:
                # Extract transcription ID and time
                match = re.search(r'Transcription (\d+) completed.*in ([\d.]+) seconds', line)
                if match:
                    tid, duration = match.groups()
                    print(f"ID {tid}: {duration}s")

Monitor API Programmatically

import requests
import time

def monitor_service():
    while True:
        try:
            response = requests.get('http://localhost:8000/health')
            health = response.json()
            print(f"Active: {health.get('active_transcriptions', 0)}")
            time.sleep(30)
        except Exception as e:
            print(f"Service down: {e}")
            time.sleep(60)

---

With comprehensive logging, you now have complete visibility into your transcription service! 🎉