bnair/VMAFOptimiser
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Refactored everything

Browse Source
This commit is contained in:
bnair
2026-01-03 11:57:26 +01:00
parent b4a82e0db5
commit f1e79ad01d
18 changed files with 2371 additions and 582 deletions
Download Patch File Download Diff File Expand all files Collapse all files

52
MAINTENANCE.md Normal file
View File

@@ -0,0 +1,52 @@
# VMAF Optimizer - Maintenance Guide
## Project Goals
To automatically optimize a large video library (TV Shows & Content) on Windows using an **AMD RX 9070 XT** GPU.
The goal is to reduce file size while maintaining "Visually Lossless" quality (VMAF 93+).
## Core Strategy
1. **Hybrid Encoding:**
* **Encode:** Use Hardware Acceleration (`av1_amf` / `hevc_amf`) for speed (critical for 2TB+ library).
* **Verify:** Use `ab-av1` (software) strictly for VMAF score calculation.
2. **Safety Net:**
* Extract a 60s sample.
* Encode sample with AV1 (QP 32).
* Calculate VMAF & Savings.
* **Decision Logic:**
* If VMAF > 97: Re-encode with higher QP (more compression) to save space.
* If VMAF < 93 or Savings < 12%: Fallback to HEVC.
* If HEVC fails: Keep original.
3. **Architecture:**
* `smart_gpu_encoder.py`: The "Engine". Handles logic, ffmpeg calls, VMAF checks. Can run standalone.
* `smart_monitor.py`: The "UI/Controller". Runs the engine in threads, displays a TUI (Rich), handles caching and directory scanning.
## Current Status
* **Working:**
* Hardware detection (AMD/NVIDIA/Intel).
* VMAF calculation (fixed regex for raw number output).
* Optimization logic (Smart QP adjustment).
* Multithreading (4 workers).
* Caching (Instant startup on second run).
* **Issues:**
* **UI Progress:** The TUI shows "Action" but Progress/Speed columns are empty.
* *Root Cause:* `smart_gpu_encoder.py` sends status updates like `Encoding AV1 (size=...)` but `smart_monitor.py` expects format `Action | Progress% | Speed`.
* **Logs:** User reported logs might be hard to find or missing if permissions fail on `Z:`.
## Debugging Instructions
To run a simplified test without scanning the whole library:
1. Create a folder `test_media` with 1 small video file.
2. Run:
```powershell
python smart_gpu_encoder.py --tv-dir ".\test_media" --content-dir ".\test_media" --jobs 1
```
## Future Todos
1. **Fix TUI Parsing:** Update `smart_gpu_encoder.py` to format strings as `Action | Percent% | Speed`.
2. **Sonarr Integration:** Add webhook triggers on success.
3. **Linux Support:** Verify `av1_qsv` or `av1_vaapi` flags for Linux/Docker deployments.
---
**Key Files:**
* `smart_gpu_encoder.py`: Core logic.
* `smart_monitor.py`: TUI and Watchdog.
* `library_cache.json`: Cache of scanned files to speed up startup.

649
README.md
View File

@@ -1,556 +1,103 @@
# VMAF Optimiser # VMAF Optimiser
Automated video library optimization to AV1 using VMAF (Video Multimethod Assessment Fusion) quality targets. Intelligently searches for optimal encoding parameters and gracefully degrades quality when needed to achieve target file size savings. **Intelligent Video Library Optimization Pipeline**
## Quick Start Automatically optimizes your video library (Movies/TV) by finding the best compression (AV1/HEVC) that maintains a high visual quality target (VMAF 93+).
### Requirements ## Features
**Prerequisites:** - **Hybrid Encoding:**
- Python 3.8+ - Uses **Hardware Acceleration** (NVIDIA NVENC, AMD AMF, Intel QSV, Apple VideoToolbox) for fast encoding.
- FFmpeg with VMAF support (`ffmpeg -filters 2>&1 | grep libvmaf`) - Uses **Software Encoding** (CPU) as a robust fallback.
- ab-av1 binary (v0.10.3+) - **VMAF Targeted:** Ensures visual quality matches the original (Target VMAF 93+).
- **Multi-Platform:** Runs on **Windows**, **Linux**, and **macOS**.
**Installation:** - **Distributed Processing:** Run multiple workers across multiple PCs sharing the same network library.
```bash - **Safety First:**
# Install ab-av1 via cargo - Locks files to prevent double-processing.
cargo install ab-av1 - Verifies output size and integrity before replacing.
- "Smart Resume" - skips already processed files.
# Or download pre-built binary
wget https://github.com/alexheretic/ab-av1/releases/download/v0.10.3/ab-av1-x86_64-unknown-linux-musl
chmod +x ab-av1
```
## Running the Optimiser
### Windows / macOS
Use the PowerShell wrapper script:
```powershell
# Interactive mode (shows prompts)
.\run_optimisation.ps1
# Direct execution with parameters
.\run_optimisation.ps1 --directory "D:\Movies" --vmaf 95 --preset 6 --workers 1
```
**Available flags:**
- `--directory <path>` - Root directory to scan (default: current directory)
- `--vmaf <score>` - Target VMAF score (default: 95.0)
- `--preset <value>` - SVT-AV1 Preset (default: 6)
- `--workers <count>` - Concurrent files to process (default: 1)
- `--samples <count>` - Samples for CRF search (default: 4)
- `--thorough` - Use thorough mode (slower, more accurate)
- `--encoder <name>` - ab-av1 encoder (default: svt-av1)
- `--hwaccel <value>` - Hardware acceleration (default: none)
### Linux / WSL
Use the bash wrapper script:
```bash
# Interactive mode
./run_optimisation.sh
# Direct execution with parameters
./run_optimisation.sh --directory /mnt/Media/Movies --vmaf 95 --workers 1
```
**Same flags as PowerShell version:**
- `--directory <path>` - Root directory to scan
- `--vmaf <score>` - Target VMAF score
- `--preset <value>` - SVT-AV1 Preset
- `--workers <count>` - Concurrent files to process
- `--samples <count>` - Samples for CRF search
- `--thorough` - Use thorough mode
- `--encoder <name>` - ab-av1 encoder
- `--hwaccel <value>` - Hardware acceleration
## How It Works
### Phase 1: Video Analysis
1. Scans directory for video files (.mkv, .mp4)
2. Uses `ffprobe` to get:
- Codec (h264, hevc, etc.)
- Resolution (width × height)
- Bitrate (calculated from size/duration)
- HDR status (color transfer detection)
3. Skips if already AV1 encoded
### Phase 2: VMAF Target Search (Intelligent Fallback)
The script tries VMAF targets in **descending order** (highest quality first):
```
Try VMAF 94 (Premium)
↓
Can achieve?
↓ Yes ↓ No
Calculate savings Try VMAF 93
↓
Savings ≥ 12%?
↓ Yes ↓ No
Encode at VMAF 94 Calculate savings
↓
Savings ≥ 12%?
↓ Yes ↓ No
Encode at VMAF 93 Find 15% (test 92, 90)
```
**Fallback Logic:**
- If VMAF 94 gives ≥12% savings → **Encode at VMAF 94**
- If VMAF 94 <12% but VMAF 93 ≥12% → **Encode at VMAF 93**
- If both <12% → Find what VMAF gives 15%+ savings:
- Tests VMAF 93, 92, 90
- Reports "FOUND 15%+ SAVINGS" with exact parameters
- Logs for manual review (no encoding)
### Phase 3: CRF Search
Uses `ab-av1 crf-search` with `--thorough` flag:
- Takes multiple samples (20-30s segments) from video
- Interpolates binary search for optimal CRF
- Outputs: Best CRF, Mean VMAF, Predicted size
- Uses `--samples` to control accuracy (default: 4 samples)
### Phase 4: Full Encoding (with Real-time Output)
If savings threshold met:
1. Runs `ab-av1 encode` with found CRF
2. **Streams all output in real-time** (you see progress live)
3. Shows ETA, encoding speed, frame count
4. Uses `--acodec copy` to preserve audio/subtitles
**Real-time output example:**
```
→ Running encoding (CRF 34)
Encoded 4320/125400 frames (3.4%)
Encoded 8640/125400 frames (6.9%)
Encoded 12960/125400 frames (10.3%)
...
Encoded 125400/125400 frames (100.0%)
Speed: 15.2 fps, ETA: 2s
```
### Phase 5: Verification & Replacement
1. Probes encoded file for actual stats
2. Calculates actual savings
3. Only replaces original if new file is smaller
4. Converts .mp4 to .mkv if needed
## Configuration
Key settings (edit in `optimize_library.py`):
```python
TARGETS = [94.0, 93.0, 92.0, 90.0] # VMAF targets to try
MIN_SAVINGS_PERCENT = 12.0 # Encode if savings ≥12%
TARGET_SAVINGS_FOR_ESTIMATE = 15.0 # Estimate for this level
PRESET = 6 # SVT-AV1 preset (4=best, 8=fast)
EXTENSIONS = {".mkv", ".mp4", ".mov", ".avi", ".ts"}
```
### What is CRF?
**Constant Rate Factor (CRF):** Quality/bitrate trade-off
- **Lower CRF** = Higher quality, larger files (e.g., CRF 20)
- **Higher CRF** = Lower quality, smaller files (e.g., CRF 40)
- **AV1 CRF range:** 0-63 (default for VMAF 94 is ~34-36)
### What is VMAF?
**Video Multimethod Assessment Fusion:** Netflix's quality metric
- **VMAF 95:** "Visually lossless" - indistinguishable from source
- **VMAF 94:** Premium quality - minor artifacts
- **VMAF 93:** Good quality - acceptable for most content
- **VMAF 90:** Standard quality - may have noticeable artifacts
- **VMAF 85:** Acceptable quality for mobile/low bandwidth
## Hardware Acceleration
**Automatic hwaccel detection:**
When `--hwaccel auto` is specified, the script selects appropriate hardware acceleration:
| Platform | Auto Selection | Notes |
|-----------|----------------|--------|
| Windows | d3d11va | Direct3D Video Acceleration |
| macOS | videotoolbox | VideoToolbox framework |
| Linux/WSL | vaapi | Video Acceleration via VA-API |
**Discrete GPU vs iGPU priority:**
- **Discrete GPU (e.g., AMD RX 7900 XT) takes priority over iGPU**
- FFmpeg/ab-av1 will prefer the more capable encoder
- For AV1 encoding, discrete GPU is selected if present
**To disable hardware acceleration:**
```powershell
.\run_optimisation.ps1 --hwaccel none
```
```bash
./run_optimisation.sh --hwaccel none
```
## Running on Multiple Machines
### Lock File Mechanism
Each video file has a corresponding lock file:
```
/opt/Optmiser/.lock/{video_filename}
```
**Process:**
1. Machine A checks for lock → None found, creates lock
2. Machine A starts encoding
3. Machine B checks for lock → Found, skips file
4. Machine A finishes, removes lock
5. Machine B can now process that file
**Result:** Different machines automatically process different files!
### Multi-Machine Setup
**Machine 1 (Linux Server - Intel i9-12900H):**
```bash
cd /opt/Optmiser
git pull origin main
./run_optimisation.sh /mnt/Media/movies --vmaf 95
```
**Machine 2 (Windows PC - AMD RX 7900 XT):**
```powershell
cd C:\Optmiser
git pull origin main
.\run_optimisation.ps1 D:\Media\movies --vmaf 95 --hwaccel auto
```
**Machine 3 (Another Linux PC):**
```bash
cd /opt/Optmiser
git pull origin main
./run_optimisation.sh /home/user/Media/tv --vmaf 95
```
All three can run simultaneously - lock files prevent duplicates!
## Logging System
All logs stored in `/opt/Optmiser/logs/` directory:
| File | Purpose |
|------|---------|
| `tv_movies.jsonl` | Successful TV & Movie encodes |
| `content.jsonl` | Successful Content folder encodes |
| `low_savings_skips.jsonl` | Files with <12% savings + 15% estimates |
| `failed_searches.jsonl` | Files that couldn't hit any VMAF target |
| `failed_encodes.jsonl` | Encoding errors |
### Log Entry Format
**Successful encode:**
```json
{
"file": "/path/to/file.mkv",
"status": "success",
"vmaf": 94.0,
"crf": 34.0,
"before": {
"codec": "h264",
"bitrate": 8500,
"size": 2684354560,
"duration": 1379.44
},
"after": {
"codec": "av1",
"bitrate": 6400,
"size": 2013265920,
"duration": 1379.44
},
"duration": 145.2,
"savings": 25.0,
"timestamp": "2025-12-31T12:00:00.000Z"
}
```
**Low savings with 15% estimate:**
```json
{
"file": "/path/to/file.mkv",
"vmaf_94": 94.0,
"savings_94": 7.0,
"vmaf_93": 93.0,
"savings_93": 18.0,
"target_for_15_percent": {
"target_vmaf": 93,
"crf": 37,
"savings": 18.0,
"quality_drop": 1,
"found": true
},
"recommendations": "logged_for_review",
"timestamp": "2025-12-31T12:00:00.000Z"
}
```
### Viewing Logs
```bash
# Watch logs in real-time
tail -f /opt/Optmiser/logs/tv_movies.jsonl | jq '.'
# Check files logged for review (both 94 and 93 <12%)
cat /opt/Optmiser/logs/low_savings_skips.jsonl | jq '.[] | select(.recommendations=="logged_for_review")'
# Statistics
jq -r '.status' /opt/Optmiser/logs/tv_movies.jsonl | sort | uniq -c
# Find what CRF/VMAF combinations are being used most
jq -r '[.vmaf, .crf] | @tsv' /opt/Optmiser/logs/tv_movies.jsonl | sort | uniq -c
```
## Troubleshooting
### Issue: "0k bitrate" display
**Cause:** VBR (Variable Bitrate) files show 0 in ffprobe's format bitrate field.
**Solution:** Calculate from `(size × 8) / duration`
### Issue: Multiple machines encoding same file
**Cause:** No coordination between machines.
**Solution:** Lock files in `/opt/Optmiser/.lock/{video_filename}`
### Issue: Encode fails with "unexpected argument"
**Cause:** Using wrong flags for ab-av1 commands.
**Solution:** Script now validates ab-av1 support at runtime and warns gracefully.
### Issue: Out of Memory
**Solution:** Reduce workers or increase swap:
```bash
# Increase swap (if needed)
sudo fallocate -l 4G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
# Use more conservative settings
./run_optimisation.sh --workers 1 --vmaf 93
```
## Best Practices
### For Servers (Intel i9-12900H)
1. **Use 50% CPU mode** if running other services (Plex, Jellyfin):
```bash
./run_optimisation.sh --workers 1 --cpu-limit 50
```
2. **Run during off-peak hours** to minimize impact on users
3. **Monitor CPU temperature**:
```bash
watch -n 2 'sensors | grep "Package id"'
```
4. **Use higher preset for faster encodes** (preset 7-8):
```bash
./run_optimisation.sh --preset 8 --vmaf 93
```
### For Windows PC (AMD RX 7900 XT)
1. **Enable hardware acceleration** for massive speedup:
```powershell
.\run_optimisation.ps1 --hwaccel auto
```
2. **Test small sample first** to verify settings:
```powershell
.\run_optimisation.ps1 --directory "D:\Media\sample" --thorough --vmaf 95
```
3. **Monitor GPU usage**:
```powershell
# Task Manager or radeontop (if available)
```
4. **Consider quality compensation:** GPU encoding may need slightly lower VMAF target (e.g., VMAF 92) to match CPU quality.
### For WSL
1. **Access Windows drives via /mnt/c/:**
```bash
ls /mnt/c/Media/movies
```
2. **Increase memory limits** if encoding 4K content:
```bash
# Edit ~/.wslconfig
[wsl2]
memory=16GB
```
## Customization
### Changing VMAF Targets
Edit `optimize_library.py`:
```python
# More aggressive (smaller files, lower quality)
TARGETS = [92.0, 90.0, 88.0]
# Conservative (larger files, higher quality)
TARGETS = [95.0, 94.0, 93.0]
```
### Changing Savings Threshold
```python
# More aggressive (encode more)
MIN_SAVINGS_PERCENT = 8.0
# Less aggressive (encode fewer)
MIN_SAVINGS_PERCENT = 15.0
```
### Changing Encoder Preset
```python
# Faster encodes (larger files, lower quality)
PRESET = 8
# Better quality (slower encodes, smaller files)
PRESET = 4
```
### Changing Estimate Target
```python
# Target higher savings for estimates
TARGET_SAVINGS_FOR_ESTIMATE = 20.0
```
## File Structure
```
/opt/Optmiser/
├── optimize_library.py # Main encoding engine
├── run_optimisation.sh # Linux/Server wrapper
├── run_optimisation.ps1 # Windows wrapper
├── bin/
│ └── ab-av1 # ab-av1 binary
├── tmp/ # Temporary encoding files
├── logs/ # Log files (JSONL format)
│ ├── tv_movies.jsonl
│ ├── content.jsonl
│ ├── low_savings_skips.jsonl
│ ├── failed_searches.jsonl
│ └── failed_encodes.jsonl
├── .lock/ # Multi-machine coordination (created at runtime)
├── README.md # This file
├── SETUP.md # Setup instructions
└── AGENTS.md # Technical documentation
```
## Platform Support Matrix
| Platform | Status | Notes |
|-----------|--------|-------|
| Linux (Intel CPU) | ✅ Supported | Software encoding, multi-worker capable |
| Windows (AMD GPU) | ✅ Supported | Hardware acceleration via d3d11va (auto-detects) |
| Windows (Intel CPU) | ✅ Supported | Software encoding |
| macOS (Apple Silicon) | ✅ Supported | Hardware via videotoolbox (auto-detects) |
| WSL (Ubuntu/Debian) | ✅ Supported | Linux compatibility layer |
| WSL (Windows drives) | ✅ Supported | Access via /mnt/c/ |
## Git Workflow
### Initial Setup
```bash
cd /opt/Optmiser
git init
git remote add origin https://gitea.theflagroup.com/bnair/VMAFOptimiser.git
git branch -M main
git add .
git commit -m "Initial commit: VMAF optimisation pipeline"
git push -u origin main
```
### Daily Updates
```bash
cd /opt/Optmiser
git pull origin main
# Run optimisation
./run_optimisation.sh /media tv_movies
# Review changes
git diff
```
### Committing Changes
```bash
cd /opt/Optmiser
git status
# Add changed files
git add optimize_library.py run_optimisation.sh run_optimisation.ps1
# Commit with message
git commit -m "feat: add Windows and Linux wrapper scripts"
# Push
git push
```
### View History
```bash
cd /opt/Optmiser
git log --oneline
git log --graph --all
```
## FAQ
**Q: Can I run this on multiple machines at once?**
A: Yes! Each machine will process different files due to lock file mechanism.
**Q: Should I use Windows or WSL?**
A: WSL is recommended for Linux compatibility. Use Windows native if you need direct hardware access or performance.
**Q: Will hardware encoding work better than CPU?**
A: For AMD RX 7900 XT, hardware AV1 encoding is ~3-10x faster than CPU. However, GPU encoding may need slightly lower VMAF targets to match quality.
**Q: What VMAF target should I use?**
A: Start with VMAF 94 or 95. Drop to 92-90 if you need more savings.
**Q: How do I know which files are being processed?**
A: Check `.lock/` directory: `ls -la /opt/Optmiser/.lock/`
**Q: Can I pause/resume?**
A: Pause by stopping the script (Ctrl+C). Resume by running again - it skips processed files.
**Q: What happens if encoding fails?**
A: Error is logged to `failed_encodes.jsonl`. Original file is NOT modified.
**Q: How much CPU does encoding use?**
A: Full CPU by default. Use `--workers 1` for single-threaded, or limit with `--cpu-limit 50` for 50% (12 threads on 24-core).
--- ---
**Last Updated:** December 31, 2025 ## Directory Structure
**Version:** 2.0 with Windows and Linux Wrapper Scripts
```
VMAFOptimiser/
├── bin/ # Place ab-av1.exe here (or install to PATH)
├── logs/ # Local logs (processed, rejected, stats)
├── locks/ # Local lock files (if network not available)
├── src/
│ ├── smart_gpu_encoder.py # Main encoder engine
│ ├── smart_monitor.py # TUI Dashboard (Watchdog)
│ ├── vmaf_common.py # Shared logic
│ └── smart_encoder.py # Legacy CPU-only engine
├── run.ps1 # Windows One-Click Start
└── README.md
```
## Setup & Installation
### 1. Requirements
- **Python 3.10+**
- **FFmpeg** (with libsvtav1/libx265 and HW drivers)
- Windows: `choco install ffmpeg`
- macOS: `brew install ffmpeg`
- Linux: `sudo apt install ffmpeg`
- **ab-av1** (VMAF calculator)
- Download from [GitHub](https://github.com/alexheretic/ab-av1) and place in `bin/` OR install via `cargo install ab-av1`
### 2. Install Python Deps
```bash
pip install watchdog rich
```
### 3. Usage
#### Interactive Dashboard (Recommended)
Monitors directories and shows a TUI with progress bars.
```bash
python src/smart_monitor.py --tv-dir "Z:\tv" --content-dir "Z:\content" --jobs 2
```
#### Headless / Background (Cron/Task Scheduler)
Runs once through the library and exits.
```bash
python src/smart_gpu_encoder.py --tv-dir "Z:\tv" --content-dir "Z:\content" --jobs 4
```
---
## Multi-PC Setup
To speed up processing, you can run this script on multiple computers simultaneously pointing to the same NAS/Network Share.
1. **Map Network Drive:** Ensure `Z:\` (or whatever path) is mapped on ALL computers.
2. **Shared Locks:** The script automatically attempts to create a `.vmaf_locks` folder in the parent directory of your TV folder (e.g., `Z:\.vmaf_locks`).
3. **Run:** Start the script on PC 1, PC 2, etc. They will respect each other's locks and not process the same file.
**Note:** Logs are stored LOCALLY on each PC in the `logs/` folder to prevent network file contention.
---
## Advanced Options
| Flag | Description |
|------|-------------|
| `--jobs N` | Number of parallel encoders (Default: 2) |
| `--tv-only` | Scan only TV Shows |
| `--content-only` | Scan only Movies |
| `--skip-until "Name"` | Skip files until this keyword is found (good for resuming) |
| `--debug` | Enable verbose logging |
---
## Credits
- Powered by `ab-av1` for VMAF calculation.
- Uses `ffmpeg` for heavy lifting.

BIN
bin/ab-av1.exe Normal file
View File

Binary file not shown.

51
debug_status.py Normal file
View File

@@ -0,0 +1,51 @@
import os
from pathlib import Path
# Paths
Z_LOGS = Path(r"Z:\.vmaf_logs")
C_LOGS = Path(r"C:\Users\bnair\Videos\encodes\logs")
LOCAL_LOGS = Path("logs").resolve()
Z_LOCKS = Path(r"Z:\.vmaf_locks")
C_LOCKS = Path(r"C:\Users\bnair\Videos\encodes\locks")
print("--- DEBUG STATUS ---")
print(f"\n1. Checking Logs on Z: ({Z_LOGS})")
if Z_LOGS.exists():
print(" [OK] Folder exists.")
for f in Z_LOGS.glob("*"):
print(f" - {f.name} ({f.stat().st_size} bytes)")
else:
print(" [MISSING] Folder does not exist.")
print(f"\n2. Checking Logs on C: ({C_LOGS})")
if C_LOGS.exists():
print(" [OK] Folder exists.")
for f in C_LOGS.glob("*"):
print(f" - {f.name} ({f.stat().st_size} bytes)")
else:
print(" [MISSING] Folder does not exist.")
print(f"\n3. Checking Local Logs: ({LOCAL_LOGS})")
if LOCAL_LOGS.exists():
print(" [OK] Folder exists.")
for f in LOCAL_LOGS.glob("*"):
print(f" - {f.name} ({f.stat().st_size} bytes)")
else:
print(" [MISSING] Folder does not exist.")
print(f"\n4. Checking Locks")
z_count = len(list(Z_LOCKS.glob("*"))) if Z_LOCKS.exists() else 0
c_count = len(list(C_LOCKS.glob("*"))) if C_LOCKS.exists() else 0
print(f" Z: Locks: {z_count}")
print(f" C: Locks: {c_count}")
print(f"\n4. Checking Temp Encodes")
temp = Path(r"C:\Users\bnair\Videos\encodes")
if temp.exists():
mkv_files = list(temp.glob("*.mkv"))
print(f" Found {len(mkv_files)} mkv files.")
for f in mkv_files:
print(f" - {f.name} ({f.stat().st_size / 1024**3:.2f} GB)")
else:
print(" [MISSING] Temp folder missing.")

207
legacy/finalOptimiser.ps1 Normal file
View File

@@ -0,0 +1,207 @@
# SMART PRE-FLIGHT ENCODER - VERBOSE CONSOLE OUTPUT
# Usage: .\Encode-TVShows.ps1
param(
[string]$TvDir = "Z:\tv",
[string]$ContentDir = "Z:\content",
[int]$MaxJobs = 2,
[int]$Av1Q = 34,
[int]$HevcQ = 28,
[string]$EncoderAV1 = "av1_amf",
[string]$EncoderHEVC = "hevc_amf",
[switch]$SkipAV1 = $true
)
# --- CONFIGURATION ---
$Global:FFMPEG = "ffmpeg"
$Global:FFPROBE = "ffprobe"
$Global:TEMP_DIR = "C:\Users\bnair\Videos\encodes"
$Global:LockDir = "C:\Users\bnair\Videos\encodes\locks"
$LogFileTV = "C:\Users\bnair\Videos\encodes\encoding-log-tv.csv"
$LogFileContent = "C:\Users\bnair\Videos\encodes\encoding-log-content.csv"
if (-not (Test-Path $Global:TEMP_DIR)) { New-Item -ItemType Directory -Path $Global:TEMP_DIR -Force | Out-Null }
if (-not (Test-Path $Global:LockDir)) { New-Item -ItemType Directory -Path $Global:LockDir -Force | Out-Null }
function Init-LogFile {
param([string]$Path)
if (-not (Test-Path $Path)) { "Timestamp,File,InputSize,OutputSize,CodecUsed,Status,Savings" | Out-File -FilePath $Path -Encoding UTF8 }
}
Init-LogFile $LogFileTV
Init-LogFile $LogFileContent
function Test-Tools {
Write-Host "Setup: Checking required tools..." -ForegroundColor Cyan
if (-not (Get-Command $Global:FFMPEG -ErrorAction SilentlyContinue)) { Write-Host "ERROR: ffmpeg not found!" -ForegroundColor Red; exit 1 }
Write-Host "Setup: Tools found." -ForegroundColor Green
}
$SharedFunctions = {
function Get-LockId {
param([string]$FilePath)
try {
$pathBytes = [System.Text.Encoding]::UTF8.GetBytes($FilePath)
$hash = [System.Security.Cryptography.SHA256]::Create().ComputeHash($pathBytes)
return [BitConverter]::ToString($hash).Replace("-", "").Substring(0, 16)
} catch { return "unknown_lock" }
}
function Run-FFmpeg {
param($In, $Out, $Enc, $Q, $Seek=$null, $Duration=$null)
$argsList = "-hide_banner -loglevel error -y"
if ($Seek) { $argsList += " -ss $Seek" }
$argsList += " -i `"$In`""
if ($Duration) { $argsList += " -t $Duration" }
$argsList += " -c:v $Enc -usage transcoding -quality quality -rc cqp -qp_i $Q -qp_p $Q -qp_b $Q -c:a copy `"$Out`""
return Start-Process -FilePath "ffmpeg" -ArgumentList $argsList -NoNewWindow -Wait -PassThru
}
function Process-VideoFile {
param($InputFile, $CurrentLogFile, $LockDir, $TempDir, $Av1Q, $HevcQ, $EncAV1, $EncHEVC, $SkipAV1)
$pidStr = $PID.ToString()
$lockId = Get-LockId -FilePath $InputFile
$lockFile = Join-Path $LockDir "$lockId.lock"
try {
if (Test-Path $lockFile) { return }
$pidStr | Out-File -FilePath $lockFile -Force
$fileName = Split-Path $InputFile -Leaf
Write-Host "[$pidStr] Found: $fileName" -ForegroundColor White
# Skip Logic
$currentCodec = (& "ffprobe" -v error -select_streams v:0 -show_entries stream=codec_name -of csv=p=0 "$InputFile" 2>&1)
if ($SkipAV1 -and ($currentCodec -match "av1" -or $currentCodec -match "hevc")) {
Write-Host "[$pidStr] SKIP: Already optimized ($currentCodec)" -ForegroundColor DarkGray
return
}
$inputSize = (Get-Item $InputFile).Length
# --- PHASE 1: PRE-FLIGHT SAMPLE ---
Write-Host "[$pidStr] Testing: Generating 60s sample..." -ForegroundColor Yellow
$tempSample = Join-Path $TempDir "$fileName.sample.mkv"
$procSample = Run-FFmpeg $InputFile $tempSample $EncAV1 $Av1Q "00:05:00" "60"
$doFullAV1 = $true
if ($procSample.ExitCode -eq 0 -and (Test-Path $tempSample)) {
$sampleSize = (Get-Item $tempSample).Length
# Threshold: 150MB for 60s is ~20Mbps (Likely bloat)
if ($sampleSize -gt 150MB) {
Write-Host "[$pidStr] Test Result: FAIL. Sample was $([math]::Round($sampleSize/1MB))MB. Too big for AV1." -ForegroundColor Red
$doFullAV1 = $false
} else {
Write-Host "[$pidStr] Test Result: PASS. Sample was $([math]::Round($sampleSize/1MB))MB. Proceeding with AV1." -ForegroundColor Green
}
Remove-Item $tempSample -Force
}
$finalStatus = "Failed"
$finalCodec = "None"
$finalSize = 0
$finalSavings = 0.00
# --- PHASE 2: FULL AV1 ---
if ($doFullAV1) {
Write-Host "[$pidStr] Action: Starting Full AV1 Encode..." -ForegroundColor Cyan
$tempAV1 = Join-Path $TempDir "$fileName.av1.mkv"
$procAV1 = Run-FFmpeg $InputFile $tempAV1 $EncAV1 $Av1Q
if ($procAV1.ExitCode -eq 0 -and (Test-Path $tempAV1)) {
$sizeAV1 = (Get-Item $tempAV1).Length
if ($sizeAV1 -lt $inputSize) {
$finalSavings = [math]::Round((1 - ($sizeAV1 / $inputSize)) * 100, 2)
Write-Host "[$pidStr] AV1 ACCEPTED: Saved ${finalSavings}%" -ForegroundColor Green
$finalOut = $InputFile -replace '\.mkv$', '_av1.mkv' -replace '\.mp4$', '_av1.mp4'
Move-Item $tempAV1 -Destination $finalOut -Force
Remove-Item $InputFile -Force
Rename-Item $finalOut -NewName $fileName -Force
"$(Get-Date -Format 'yyyy-MM-dd HH:mm:ss'),`"$InputFile`",$inputSize,$sizeAV1,AV1,`"Replaced (AV1)`",$finalSavings%" | Out-File -FilePath $CurrentLogFile -Append -Encoding UTF8
return
} else {
Write-Host "[$pidStr] AV1 REJECTED: Larger than source ($([math]::Round($sizeAV1/1GB,2)) GB). Deleting..." -ForegroundColor Red
Remove-Item $tempAV1 -Force
}
}
}
# --- PHASE 3: HEVC FALLBACK ---
Write-Host "[$pidStr] Action: Trying HEVC Fallback..." -ForegroundColor Cyan
$tempHEVC = Join-Path $TempDir "$fileName.hevc.mkv"
$procHEVC = Run-FFmpeg $InputFile $tempHEVC $EncHEVC $HevcQ
if ($procHEVC.ExitCode -eq 0 -and (Test-Path $tempHEVC)) {
$sizeHEVC = (Get-Item $tempHEVC).Length
if ($sizeHEVC -lt $inputSize) {
$finalSavings = [math]::Round((1 - ($sizeHEVC / $inputSize)) * 100, 2)
Write-Host "[$pidStr] HEVC ACCEPTED: Saved ${finalSavings}%" -ForegroundColor Green
$finalOut = $InputFile -replace '\.mkv$', '_hevc.mkv' -replace '\.mp4$', '_hevc.mp4'
Move-Item $tempHEVC -Destination $finalOut -Force
Remove-Item $InputFile -Force
Rename-Item $finalOut -NewName $fileName -Force
$finalStatus = "Replaced (HEVC)"
$finalCodec = "HEVC"
$finalSize = $sizeHEVC
} else {
Write-Host "[$pidStr] HEVC REJECTED: Also larger. Keeping original." -ForegroundColor Red
Remove-Item $tempHEVC -Force
$finalStatus = "Rejected (Both Larger)"
$finalSize = $sizeHEVC
$finalCodec = "HEVC"
}
}
"$(Get-Date -Format 'yyyy-MM-dd HH:mm:ss'),`"$InputFile`",$inputSize,$finalSize,$finalCodec,`"$finalStatus`",$finalSavings%" | Out-File -FilePath $CurrentLogFile -Append -Encoding UTF8
} finally {
Remove-Item $lockFile -Force -ErrorAction SilentlyContinue
}
}
}
function Process-Directory {
param($TargetDirectory, $TargetLogFile, $PhaseName)
if (-not (Test-Path $TargetDirectory)) { return }
Write-Host "`n=== PHASE: $PhaseName ===" -ForegroundColor Magenta
$files = Get-ChildItem -Path $TargetDirectory -Include *.mkv, *.mp4 -Recurse -File -ErrorAction SilentlyContinue
$videoFiles = $files | Where-Object { $_.Name -notmatch "_av1" -and $_.Name -notmatch "_hevc" }
Write-Host "Found $($videoFiles.Count) files." -ForegroundColor Cyan
$processed = 0
while ($processed -lt $videoFiles.Count) {
$batchSize = [math]::Min($MaxJobs, ($videoFiles.Count - $processed))
$currentBatch = $videoFiles[$processed..($processed + $batchSize - 1)]
$jobs = @()
foreach ($file in $currentBatch) {
$jobs += Start-Job -InitializationScript $SharedFunctions -ScriptBlock {
param($f, $log, $lock, $temp, $av1q, $hevcq, $e1, $e2, $skip)
Process-VideoFile $f $log $lock $temp $av1q $hevcq $e1 $e2 $skip
} -ArgumentList $file.FullName, $TargetLogFile, $Global:LockDir, $Global:TEMP_DIR, $Av1Q, $HevcQ, $EncoderAV1, $EncoderHEVC, $SkipAV1
}
while (($jobs | Where-Object { $_.State -eq 'Running' }).Count -gt 0) {
$jobs | Receive-Job
Start-Sleep -Seconds 2
}
$jobs | Receive-Job
$jobs | Remove-Job -Force
$processed += $batchSize
Write-Host "Progress Phase ${PhaseName}: $processed / $($videoFiles.Count)" -ForegroundColor Yellow
}
}
Test-Tools
Process-Directory $TvDir $LogFileTV "TV-Shows"
Process-Directory $ContentDir $LogFileContent "Content"
Write-Host "`nDone." -ForegroundColor Green

69
optimize_library.py → legacy/optimize_library.py
View File

@@ -150,7 +150,15 @@ def get_probe_data(filepath):
"-show_format", "-show_format",
str(filepath), str(filepath),
] ]
res = subprocess.run(cmd, capture_output=True, text=True, check=True) # FIX: Added encoding="utf-8" and errors="ignore"
res = subprocess.run(
cmd,
capture_output=True,
text=True,
check=True,
encoding="utf-8",
errors="ignore"
)
return json.loads(res.stdout) return json.loads(res.stdout)
except Exception as e: except Exception as e:
print(f"Error probing {filepath}: {e}") print(f"Error probing {filepath}: {e}")
@@ -235,6 +243,7 @@ def run_crf_search(
target_vmaf + HW_ENCODER_VMAF_OFFSET if is_hw_encoder else target_vmaf target_vmaf + HW_ENCODER_VMAF_OFFSET if is_hw_encoder else target_vmaf
) )
# --- Build the Command (Used for both running and parsing) ---
cmd = [ cmd = [
"ab-av1", "ab-av1",
"crf-search", "crf-search",
@@ -252,10 +261,9 @@ def run_crf_search(
"4", "4",
] ]
# Add encoder if not default # FORCE use of the specified encoder (Fixes the CPU default crash)
if encoder != "svt-av1": if ab_av1_supports("crf-search", "--encoder"):
if ab_av1_supports("crf-search", "--encoder"): cmd.extend(["--encoder", encoder])
cmd.extend(["--encoder", encoder])
# Hardware decode acceleration # Hardware decode acceleration
if use_hw and hwaccel: if use_hw and hwaccel:
@@ -266,27 +274,19 @@ def run_crf_search(
vmaf_label = f"VMAF {effective_vmaf}" if is_hw_encoder else f"VMAF {target_vmaf}" vmaf_label = f"VMAF {effective_vmaf}" if is_hw_encoder else f"VMAF {target_vmaf}"
print(f" - Searching for CRF to hit {vmaf_label}...") print(f" - Searching for CRF to hit {vmaf_label}...")
# 1. First Run: Stream output to console
returncode = run_command_streaming(cmd, f"crf-search {vmaf_label}") returncode = run_command_streaming(cmd, f"crf-search {vmaf_label}")
if returncode == 0: if returncode == 0:
# Parse output to find CRF and predicted size # 2. Second Run: Capture output to parse the CRF
# FIX: Reuse 'cmd' and add encoding parameters to prevent charmap crashes
res = subprocess.run( res = subprocess.run(
[ cmd,
"ab-av1",
"crf-search",
"-i",
str(filepath),
"--min-vmaf",
str(target_vmaf),
"--preset",
str(preset),
"--temp-dir",
temp_dir,
"--samples",
"4",
],
capture_output=True, capture_output=True,
text=True, text=True,
encoding="utf-8",
errors="ignore"
) )
lines = res.stdout.strip().split("\n") lines = res.stdout.strip().split("\n")
@@ -463,13 +463,18 @@ def process_file(
"""Process a single video file with intelligent VMAF targeting""" """Process a single video file with intelligent VMAF targeting"""
global _shutdown_requested global _shutdown_requested
# Determine if THIS worker should use hardware encoder # --- LOGIC FIX START ---
use_hw = False # If "use_hw_mode" (Hybrid) is FALSE, use the requested encoder for EVERYONE.
if use_hw_mode and hwaccel and hw_encoder in HW_ENCODERS: if not use_hw_mode:
use_hw = claim_hardware_worker() encoder = hw_encoder
use_hw = (encoder in HW_ENCODERS)
# HW worker uses hardware encoder; CPU workers use svt-av1 else:
encoder = hw_encoder if use_hw else "svt-av1" # Hybrid Mode: Only 1 worker gets HW, rest get CPU
use_hw = False
if hwaccel and hw_encoder in HW_ENCODERS:
use_hw = claim_hardware_worker()
encoder = hw_encoder if use_hw else "svt-av1"
# --- LOGIC FIX END ---
filepath = Path(filepath) filepath = Path(filepath)
lock_file = Path(log_dir).parent / ".lock" / f"{filepath.name}.lock" lock_file = Path(log_dir).parent / ".lock" / f"{filepath.name}.lock"
@@ -502,9 +507,13 @@ def process_file(
_processed_files.add(file_key) _processed_files.add(file_key)
print(f"\n--- Processing: {filepath.name} ---") print(f"\n--- Processing: {filepath.name} ---")
print( # Added 'errors="ignore"' to print to avoid Unicode crashes on console
f" Source: {stats_before['codec']} @ {stats_before['bitrate']}k, {stats_before['size'] / (1024**3):.2f} GB" try:
) print(
f" Source: {stats_before['codec']} @ {stats_before['bitrate']}k, {stats_before['size'] / (1024**3):.2f} GB"
)
except UnicodeEncodeError:
print(f" Source: {stats_before['codec']} @ {stats_before['bitrate']}k")
if _shutdown_requested: if _shutdown_requested:
return False return False

2
run_optimisation.ps1 → legacy/run_optimisation.ps1
View File

@@ -28,7 +28,7 @@ function Invoke-OptimizeLibrary {
exit 1 exit 1
} }
$pythonCmd = Get-Command python3, python, py -ErrorAction SilentlyContinue | Select-Object -First 1 $pythonCmd = Get-Command "python"
if (-not $pythonCmd) { if (-not $pythonCmd) {
Write-ColorOutput -Message "ERROR: Python 3 not found. Please install Python 3." -Color "Red" Write-ColorOutput -Message "ERROR: Python 3 not found. Please install Python 3." -Color "Red"
exit 1 exit 1

0
run_optimisation.sh → legacy/run_optimisation.sh
View File

30
legacy/run_smart_encoder.ps1 Normal file
View File

@@ -0,0 +1,30 @@
# Run Smart GPU Encoder
# Wrapper to ensure correct environment
$ScriptPath = "$PSScriptRoot\smart_gpu_encoder.py"
$PythonPath = "python" # Or specific path like "C:\Python39\python.exe"
# Directories (Change these to your actual paths)
$TvDir = "Z:\tv"
$ContentDir = "Z:\content"
Write-Host "==========================================" -ForegroundColor Cyan
Write-Host " SMART GPU ENCODER (AMD AMF + VMAF)" -ForegroundColor Cyan
Write-Host "==========================================" -ForegroundColor Cyan
Write-Host "Script: $ScriptPath"
Write-Host "TV Dir: $TvDir"
Write-Host "Content Dir: $ContentDir"
Write-Host ""
# Check if ab-av1 exists in bin
if (-not (Test-Path "$PSScriptRoot\bin\ab-av1.exe")) {
Write-Host "WARNING: ab-av1.exe not found in bin folder!" -ForegroundColor Red
Write-Host "Please download it and place it in $PSScriptRoot\bin\"
exit 1
}
# Run the python script
& $PythonPath $ScriptPath --tv-dir $TvDir --content-dir $ContentDir
Write-Host "`nDone." -ForegroundColor Green
Read-Host "Press Enter to exit..."

1
library_cache.json Normal file
View File

File diff suppressed because one or more lines are too long

12
run.ps1 Normal file
View File

@@ -0,0 +1,12 @@
# Run Smart GPU Encoder (Wrapper)
$ScriptPath = "$PSScriptRoot\smart_gpu_encoder.py"
$PythonPath = "python"
# Default to interactive mode if no args
if ($args.Count -eq 0) {
Write-Host "Starting Smart Encoder..." -ForegroundColor Cyan
& $PythonPath $ScriptPath
} else {
# Pass through arguments
& $PythonPath $ScriptPath $args
}

191
run_smart_encoder.sh Normal file
View File

@@ -0,0 +1,191 @@
#!/bin/bash
# Smart Video Encoder Launcher - Linux/WSL
# Usage: ./run_smart_encoder.sh [options]
set -e
# Colors
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
CYAN='\033[0;36m'
MAGENTA='\033[0;35m'
RESET='\033[0m'
# Configuration
TV_DIR="${TV_DIR:-/mnt/z/tv}"
CONTENT_DIR="${CONTENT_DIR:-/mnt/z/content}"
JOBS="${JOBS:-1}"
TV_ONLY=false
CONTENT_ONLY=false
# Parse arguments
while [[ $# -gt 0 ]]; do
case "$1" in
--tv-dir)
TV_DIR="$2"
shift 2
;;
--content-dir)
CONTENT_DIR="$2"
shift 2
;;
--jobs)
JOBS="$2"
shift 2
;;
--tv-only)
TV_ONLY=true
shift
;;
--content-only)
CONTENT_ONLY=true
shift
;;
--help|-h)
echo "Usage: $0 [options]"
echo ""
echo "Options:"
echo " --tv-dir <path> TV directory (default: /mnt/z/tv)"
echo " --content-dir <path> Content directory (default: /mnt/z/content)"
echo " --jobs <count> Parallel jobs (default: 1)"
echo " --tv-only Process TV directory only"
echo " --content-only Process content directory only"
echo " --help, -h Show this help"
exit 0
;;
*)
echo "Unknown option: $1"
echo "Use --help for usage"
exit 1
;;
esac
done
log_info() {
echo -e "${CYAN}$*${RESET}"
}
log_success() {
echo -e "${GREEN}✓ $*${RESET}"
}
log_error() {
echo -e "${RED}❌ $*${RESET}"
}
log_warn() {
echo -e "${YELLOW}⚠️ $*${RESET}"
}
# Get script directory
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PYTHON_SCRIPT="$SCRIPT_DIR/smart_encoder.py"
# Check dependencies
log_info "Checking dependencies..."
if ! command -v python3 &> /dev/null; then
log_error "Python 3 not found"
log_warn "Install with: sudo apt install python3"
exit 1
fi
if ! command -v ffmpeg &> /dev/null; then
log_error "FFmpeg not found"
log_warn "Install with: sudo apt install ffmpeg"
exit 1
fi
if ! command -v ffprobe &> /dev/null; then
log_error "FFprobe not found"
log_warn "Install with: sudo apt install ffmpeg"
exit 1
fi
if ! command -v ab-av1 &> /dev/null; then
log_error "ab-av1 not found"
log_warn "Install with: cargo install ab-av1"
exit 1
fi
log_success "All dependencies found"
# Check if script exists
if [[ ! -f "$PYTHON_SCRIPT" ]]; then
log_error "smart_encoder.py not found at: $PYTHON_SCRIPT"
exit 1
fi
# Make script executable
chmod +x "$PYTHON_SCRIPT"
# Print banner
echo ""
echo -e "${MAGENTA}============================================================${RESET}"
echo -e "${MAGENTA}🎬 Smart Video Encoder - Linux/WSL${RESET}"
echo -e "${MAGENTA}============================================================${RESET}"
echo ""
# Print configuration
log_info "Configuration:"
echo " TV Directory: $TV_DIR"
echo " Content Directory: $CONTENT_DIR"
echo " Parallel Jobs: $JOBS"
if [[ "$TV_ONLY" == true ]]; then
echo -e " ${YELLOW}Mode: TV only${RESET}"
elif [[ "$CONTENT_ONLY" == true ]]; then
echo -e " ${YELLOW}Mode: Content only${RESET}"
else
echo -e " ${YELLOW}Mode: TV + Content${RESET}"
fi
# Check directories
if [[ "$TV_ONLY" == false ]] && [[ ! -d "$TV_DIR" ]]; then
log_warn "TV directory not found: $TV_DIR"
fi
if [[ "$CONTENT_ONLY" == false ]] && [[ ! -d "$CONTENT_DIR" ]]; then
log_warn "Content directory not found: $CONTENT_DIR"
fi
# Build command
CMD="python3 $PYTHON_SCRIPT --tv-dir $TV_DIR --content-dir $CONTENT_DIR --jobs $JOBS"
if [[ "$TV_ONLY" == true ]]; then
CMD="$CMD --tv-only"
elif [[ "$CONTENT_ONLY" == true ]]; then
CMD="$CMD --content-only"
fi
# Run
echo ""
log_info "Starting encoder..."
echo -e "${CYAN}============================================================${RESET}"
echo ""
eval $CMD
EXIT_CODE=$?
echo ""
echo -e "${CYAN}============================================================${RESET}"
if [[ $EXIT_CODE -eq 0 ]]; then
log_success "Encoding completed successfully"
else
log_error "Encoding failed with exit code: $EXIT_CODE"
fi
echo -e "${CYAN}============================================================${RESET}"
# Show log locations
echo ""
log_info "Log files:"
echo " $HOME/Videos/encodes/logs/tv.jsonl"
echo " $HOME/Videos/encodes/logs/content.jsonl"
echo " $HOME/Videos/encodes/logs/rejected.jsonl"
echo " $HOME/Videos/encodes/logs/metadata.jsonl"
echo ""
exit $EXIT_CODE

444
src/smart_encoder.py Normal file
View File

@@ -0,0 +1,444 @@
#!/usr/bin/env python3
"""
Smart Video Encoder - AV1 with HEVC Fallback
Runs on Windows with native ab-av1.exe and ffmpeg
Refactored to use vmaf_common.py
"""
import os
import sys
import subprocess
import json
import shutil
import time
import argparse
import signal
import platform
import threading
from pathlib import Path
from concurrent.futures import ThreadPoolExecutor, as_completed
import vmaf_common as common
# --- Configuration ---
TARGET_VMAF_MIN = common.DEFAULT_CONFIG["target_vmaf"]
MIN_SAVINGS_PERCENT = common.DEFAULT_CONFIG["min_savings"]
MAX_JOBS = common.DEFAULT_CONFIG["cpu_jobs"]
AV1_QUALITY = common.DEFAULT_CONFIG["av1_crf"]
HEVC_QUALITY = common.DEFAULT_CONFIG["hevc_crf"]
SAMPLE_DURATION = 60
SAMPLE_START = 300
MAX_SAMPLE_SIZE_MB = 150
TEMP_DIR = common.get_temp_dir()
# Tools
FFMPEG_BIN = "ffmpeg"
AB_AV1_BIN = "ab-av1"
# Global state
_shutdown_requested = False
_lock_files = {}
def signal_handler(signum, frame):
global _shutdown_requested
_shutdown_requested = True
print("\n\n[WARNING] Shutdown requested. Finishing current tasks...")
if platform.system() != "Windows":
signal.signal(signal.SIGINT, signal_handler)
signal.signal(signal.SIGTERM, signal_handler)
def run_command_streaming(cmd, description=""):
"""Run command and stream output in real-time"""
print(f" ▶ Running: {description}")
process = subprocess.Popen(
cmd,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
universal_newlines=True,
bufsize=1
)
output_lines = []
if process.stdout:
for line in process.stdout:
if _shutdown_requested:
process.terminate()
break
line = line.rstrip()
print(f" {line}")
output_lines.append(line)
process.wait()
return process.returncode, "\n".join(output_lines)
def test_av1_sample(filepath, output_path):
"""Test AV1 encoding on a sample segment"""
print(f" 🧪 Testing AV1 with {SAMPLE_DURATION}s sample...")
cmd = [
"ffmpeg",
"-hide_banner",
"-loglevel", "warning",
"-ss", str(SAMPLE_START),
"-i", str(filepath),
"-t", str(SAMPLE_DURATION),
"-c:v", "libsvtav1",
"-crf", str(AV1_QUALITY),
"-preset", "6",
"-c:a", "copy",
"-c:s", "copy",
"-y",
str(output_path)
]
returncode, output = run_command_streaming(cmd, f"AV1 sample test")
return returncode == 0 and output_path.exists()
def encode_av1_full(filepath, output_path):
"""Full AV1 encode using ab-av1"""
print(f" 🎬 Full AV1 encode...")
cmd = [
"ab-av1", "encode",
"-i", str(filepath),
"-o", str(output_path),
"--crf", str(AV1_QUALITY),
"--preset", "6",
"--acodec", "copy"
]
return run_command_streaming(cmd, f"AV1 full encode")[0] == 0
def encode_hevc_full(filepath, output_path):
"""Full HEVC encode using ffmpeg"""
print(f" 🎬 Full HEVC encode...")
cmd = [
"ffmpeg",
"-hide_banner",
"-loglevel", "warning",
"-i", str(filepath),
"-c:v", "libx265",
"-crf", str(HEVC_QUALITY),
"-preset", "medium",
"-c:a", "copy",
"-c:s", "copy",
"-y",
str(output_path)
]
return run_command_streaming(cmd, f"HEVC full encode")[0] == 0
def process_file(filepath, log_dir, log_category, lock_dir):
"""Process a single video file with AV1→HEVC fallback"""
filepath = Path(filepath)
filename = filepath.name
# Check lock (using shared logic now)
lock_file = common.acquire_lock(lock_dir, filepath)
if not lock_file:
print(f" 🔒 Skipping (locked): {filename}")
return True
try:
print(f"\n{'='*60}")
print(f"📁 Processing: {filename}")
print(f"{'='*60}")
# Get initial metadata
metadata_before = common.get_video_info(filepath)
if not metadata_before:
print(f"❌ Could not read metadata, skipping")
return False
print(f" 📊 Original:")
print(f" Codec: {metadata_before['codec']}")
print(f" Size: {metadata_before['size'] / (1024**3):.2f} GB")
print(f" Bitrate: {metadata_before['bitrate']} kbps")
print(f" Duration: {metadata_before['duration'] / 60:.1f} min")
# Skip if already AV1 or HEVC
if metadata_before['codec'] in ['av1', 'hevc']:
print(f" ℹ️ Already optimized ({metadata_before['codec']}), skipping")
# We don't log skips to main log to avoid clutter, but could if needed
return True
input_size = metadata_before['size']
# --- PHASE 1: AV1 SAMPLE TEST ---
sample_output = TEMP_DIR / f"{filepath.stem}.sample.mkv"
use_av1 = True
try:
av1_test_passed = test_av1_sample(filepath, sample_output)
if av1_test_passed:
sample_size = sample_output.stat().st_size
sample_size_mb = sample_size / (1024 * 1024)
print(f" 📏 Sample size: {sample_size_mb:.1f} MB")
# Extrapolate to full file size
duration_ratio = metadata_before['duration'] / SAMPLE_DURATION
estimated_full_size = sample_size * duration_ratio
estimated_full_gb = estimated_full_size / (1024**3)
input_gb = input_size / (1024**3)
print(f" 📈 Estimated full size: {estimated_full_gb:.2f} GB")
print(f" 📉 Original size: {input_gb:.2f} GB")
if sample_size_mb > MAX_SAMPLE_SIZE_MB:
print(f" ❌ AV1 REJECTED: Sample too large ({sample_size_mb:.1f} MB > {MAX_SAMPLE_SIZE_MB} MB)")
use_av1 = False
elif estimated_full_size >= input_size:
print(f" ❌ AV1 REJECTED: Estimated size ({estimated_full_gb:.2f} GB) >= original ({input_gb:.2f} GB)")
use_av1 = False
else:
estimated_savings = (1 - estimated_full_size / input_size) * 100
print(f" ✅ AV1 PASS: Estimated savings {estimated_savings:.1f}%")
else:
print(f" ❌ AV1 sample test failed")
use_av1 = False
if sample_output.exists():
sample_output.unlink()
except Exception as e:
print(f" ❌ AV1 test error: {e}")
use_av1 = False
# --- PHASE 2: ENCODE ---
temp_output = TEMP_DIR / f"{filepath.stem}.temp.mkv"
final_status = "rejected"
final_codec = None
final_size = input_size
final_savings = 0.0
if use_av1:
# Try AV1 full encode
if encode_av1_full(filepath, temp_output):
metadata_after = common.get_video_info(temp_output)
if metadata_after:
final_size = metadata_after['size']
final_savings = (1 - final_size / input_size) * 100
if final_size < input_size:
final_status = "success"
final_codec = "av1"
print(f" ✅ AV1 SUCCESS: Saved {final_savings:.1f}%")
else:
print(f" ❌ AV1 FAILED: Final size >= original")
if temp_output.exists(): temp_output.unlink()
# Fall back to HEVC
print(f" 🔄 Trying HEVC fallback...")
if encode_hevc_full(filepath, temp_output):
metadata_after = common.get_video_info(temp_output)
if metadata_after:
final_size = metadata_after['size']
final_savings = (1 - final_size / input_size) * 100
if final_size < input_size:
final_status = "success"
final_codec = "hevc"
print(f" ✅ HEVC SUCCESS: Saved {final_savings:.1f}%")
else:
print(f" ❌ HEVC FAILED: Also larger than original")
if temp_output.exists(): temp_output.unlink()
else:
print(f" ❌ AV1 encode failed, trying HEVC...")
if encode_hevc_full(filepath, temp_output):
metadata_after = common.get_video_info(temp_output)
if metadata_after:
final_size = metadata_after['size']
final_savings = (1 - final_size / input_size) * 100
if final_size < input_size:
final_status = "success"
final_codec = "hevc"
print(f" ✅ HEVC SUCCESS: Saved {final_savings:.1f}%")
else:
print(f" ❌ HEVC FAILED: Larger than original")
if temp_output.exists(): temp_output.unlink()
else:
# AV1 test failed, try HEVC directly
print(f" 🔄 Trying HEVC directly...")
if encode_hevc_full(filepath, temp_output):
metadata_after = common.get_video_info(temp_output)
if metadata_after:
final_size = metadata_after['size']
final_savings = (1 - final_size / input_size) * 100
if final_size < input_size:
final_status = "success"
final_codec = "hevc"
print(f" ✅ HEVC SUCCESS: Saved {final_savings:.1f}%")
else:
print(f" ❌ HEVC FAILED: Larger than original")
if temp_output.exists(): temp_output.unlink()
# --- PHASE 3: FINALIZE ---
if final_status == "success":
# Replace original file (Safe Upload)
if filepath.suffix:
backup_path = filepath.with_suffix(f"{filepath.suffix}.backup")
else:
backup_path = Path(str(filepath) + ".backup")
shutil.move(str(filepath), str(backup_path))
shutil.move(str(temp_output), str(filepath))
# Verify Integrity
if Path(filepath).stat().st_size == final_size:
backup_path.unlink()
metadata_after = common.get_video_info(filepath)
common.log_event(log_dir, f"{log_category}.jsonl", {
"file": str(filepath),
"status": "success",
"codec": final_codec,
"input_size": input_size,
"output_size": final_size,
"savings_percent": final_savings,
"metadata_before": metadata_before,
"metadata_after": metadata_after
})
print(f"\n ✅ SUCCESS: Optimized with {final_codec.upper() if final_codec else 'unknown'}")
print(f" Savings: {final_savings:.1f}% ({input_size / (1024**3):.2f} GB → {final_size / (1024**3):.2f} GB)")
else:
print(" ❌ Critical Error: Copied file size mismatch! Restoring backup.")
shutil.move(str(backup_path), str(filepath))
else:
common.log_event(log_dir, "rejected.jsonl", {
"file": str(filepath),
"status": "rejected",
"reason": "both_codecs_larger_than_original",
"input_size": input_size,
"metadata": metadata_before
})
print(f"\n ❌ REJECTED: Both AV1 and HEVC larger than original")
print(f" Keeping original file ({input_size / (1024**3):.2f} GB)")
return True
except Exception as e:
print(f"❌ Error processing {filename}: {e}")
return False
finally:
# Release shared lock
if lock_file and lock_file.exists():
lock_file.unlink()
def scan_directory(directory, extensions=None):
"""Scan directory for video files"""
if extensions is None:
extensions = {".mkv", ".mp4", ".mov", ".avi", ".ts"}
files = []
for dirpath, dirnames, filenames in os.walk(directory):
# Skip processed/system files
dirnames[:] = [d for d in dirnames if not d.startswith("_") and d not in [".recycle", "@eaDir"]]
for filename in filenames:
filepath = Path(dirpath) / filename
if filepath.suffix.lower() in extensions:
if "_av1" in filepath.stem.lower() or "_hevc" in filepath.stem.lower():
continue
files.append(filepath)
return sorted(files)
def main():
parser = argparse.ArgumentParser(description="Smart Video Encoder - AV1 with HEVC Fallback")
parser.add_argument("--tv-dir", default=common.DEFAULT_CONFIG["tv_dir"], help="TV directory")
parser.add_argument("--content-dir", default=common.DEFAULT_CONFIG["content_dir"], help="Content directory")
parser.add_argument("--jobs", type=int, default=MAX_JOBS, help="Parallel jobs")
parser.add_argument("--tv-only", action="store_true", help="Process TV only")
parser.add_argument("--content-only", action="store_true", help="Process content only")
args = parser.parse_args()
print("="*60)
print("🎬 Smart Video Encoder - AV1 with HEVC Fallback")
print("="*60)
# Setup
common.check_dependencies()
lock_dir, log_dir = common.get_base_paths(args)
TEMP_DIR.mkdir(parents=True, exist_ok=True)
print(f"\n📁 Configuration:")
print(f" TV Directory: {args.tv_dir}")
print(f" Content Directory: {args.content_dir}")
print(f" Parallel Jobs: {args.jobs}")
print(f" Locks: {lock_dir}")
print(f" Logs: {log_dir}")
print()
tasks = []
if not args.content_only:
tv_dir = Path(args.tv_dir)
if tv_dir.exists():
tv_files = scan_directory(tv_dir)
print(f"📺 TV Files: {len(tv_files)}")
for f in tv_files:
tasks.append((f, log_dir, "tv_shows", lock_dir))
if not args.tv_only:
content_dir = Path(args.content_dir)
if content_dir.exists():
content_files = scan_directory(content_dir)
print(f"📦 Content Files: {len(content_files)}")
for f in content_files:
tasks.append((f, log_dir, "content", lock_dir))
if not tasks:
print("❌ No files to process")
return
print(f"\n🚀 Processing {len(tasks)} files...")
print("="*60)
success_count = 0
fail_count = 0
with ThreadPoolExecutor(max_workers=args.jobs) as executor:
futures = {}
for item in tasks:
# item = (filepath, log_dir, log_category, lock_dir)
future = executor.submit(process_file, *item)
futures[future] = item[0]
for future in as_completed(futures):
if _shutdown_requested:
break
filepath = futures[future]
try:
result = future.result()
if result:
success_count += 1
else:
fail_count += 1
except Exception as e:
print(f"❌ Error processing {filepath}: {e}")
fail_count += 1
print("\n" + "="*60)
print("📊 Summary:")
print(f" Processed: {success_count + fail_count}")
print(f" ✅ Success: {success_count}")
print(f" ❌ Failed: {fail_count}")
print("="*60)
if __name__ == "__main__":
main()

504
src/smart_gpu_encoder.py Normal file
View File

@@ -0,0 +1,504 @@
#!/usr/bin/env python3
"""
Smart GPU Encoder (Windows/AMD/NVIDIA Optimized)
------------------------------------------------
Refactored to use vmaf_common.py
"""
import os
import sys
import subprocess
import json
import shutil
import time
import argparse
import signal
import platform
import re
import threading
from pathlib import Path
from datetime import datetime
from concurrent.futures import ThreadPoolExecutor, as_completed
# --- Import Common Module ---
import vmaf_common as common
# --- Configuration ---
TARGET_VMAF_MIN = common.DEFAULT_CONFIG["target_vmaf"]
MIN_SAVINGS_PERCENT = common.DEFAULT_CONFIG["min_savings"]
MAX_JOBS = common.DEFAULT_CONFIG["gpu_jobs"]
DEFAULT_AV1_QP = 32
DEFAULT_HEVC_QP = 28
SAMPLE_DURATION = 60
SAMPLE_START_TIME = 300
TEMP_DIR = None # Will be set in main
# Tools (Local override if needed, else used from common checks)
FFMPEG_BIN = "ffmpeg"
AB_AV1_BIN = "ab-av1"
# Global state
shutdown_requested = False
active_processes = set()
upload_lock = threading.Lock()
proc_lock = threading.Lock()
debug_mode = False
def handle_sigint(signum, frame):
global shutdown_requested
print("\n\n[!] CRITICAL: Shutdown requested (Ctrl+C).")
print(" Killing active encoder processes...")
shutdown_requested = True
with proc_lock:
for proc in list(active_processes):
try:
proc.terminate()
time.sleep(0.1)
if proc.poll() is None:
proc.kill()
except:
pass
print(" Cleanup complete. Exiting.")
sys.exit(1)
signal.signal(signal.SIGINT, handle_sigint)
# --- Hardware Detection ---
def detect_hardware_encoder():
"""Detects available hardware encoders via ffmpeg"""
try:
res = subprocess.run([FFMPEG_BIN, "-hide_banner", "-encoders"], capture_output=True, text=True)
out = res.stdout
av1_enc = None
hevc_enc = None
# Check AMD
if "av1_amf" in out: av1_enc = "av1_amf"
if "hevc_amf" in out: hevc_enc = "hevc_amf"
if av1_enc or hevc_enc: return av1_enc, hevc_enc, "amf"
# Check NVIDIA
if "av1_nvenc" in out: av1_enc = "av1_nvenc"
if "hevc_nvenc" in out: hevc_enc = "hevc_nvenc"
if av1_enc or hevc_enc: return av1_enc, hevc_enc, "nvenc"
# Check Intel
if "av1_qsv" in out: av1_enc = "av1_qsv"
if "hevc_qsv" in out: hevc_enc = "hevc_qsv"
if av1_enc or hevc_enc: return av1_enc, hevc_enc, "qsv"
# Check Apple
if "av1_videotoolbox" in out: av1_enc = "av1_videotoolbox"
if "hevc_videotoolbox" in out: hevc_enc = "hevc_videotoolbox"
if av1_enc or hevc_enc: return av1_enc, hevc_enc, "videotoolbox"
return None, None, "cpu"
except Exception as e:
print(f"[Warning] HW Detection failed: {e}")
return None, None, "cpu"
def get_encoder_args(codec, encoder, qp):
"""Returns correct ffmpeg args for specific HW vendor"""
if not encoder: return []
# AMD AMF
if "amf" in encoder:
common_args = ["-rc", "cqp", "-qp_i", str(qp), "-qp_p", str(qp), "-qp_b", str(qp), "-quality", "quality"]
return ["-c:v", encoder, "-usage", "transcoding"] + common_args
# NVIDIA NVENC
if "nvenc" in encoder:
return ["-c:v", encoder, "-rc", "constqp", "-qp", str(qp), "-preset", "p6", "-spatial-aq", "1"]
# Intel QSV
if "qsv" in encoder:
return ["-c:v", encoder, "-global_quality", str(qp), "-look_ahead", "1"]
# Apple VideoToolbox
if "videotoolbox" in encoder:
q = int(100 - (qp * 2))
return ["-c:v", encoder, "-q:v", str(q)]
# Software Fallback
if encoder == "libsvtav1":
# CRF 20-35 range usually good
return ["-c:v", "libsvtav1", "-crf", str(qp), "-preset", "6", "-g", "240"]
if encoder == "libx265":
return ["-c:v", "libx265", "-crf", str(qp), "-preset", "medium"]
return []
# --- Helpers ---
def run_process(cmd, description="", status_callback=None):
"""Run a process with real-time output and clean shutdown tracking"""
if shutdown_requested: return False
if status_callback: status_callback(description)
try:
# Windows: Hide console window
cflags = 0x08000000 if platform.system() == 'Windows' else 0
proc = subprocess.Popen(
cmd,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
universal_newlines=True,
bufsize=1,
creationflags=cflags
)
with proc_lock:
active_processes.add(proc)
if proc.stdout:
for line in proc.stdout:
if shutdown_requested:
proc.terminate()
break
line = line.strip()
if line:
if debug_mode: print(f" [Debug] {line}")
if status_callback and ("frame=" in line or "size=" in line or "time=" in line):
status_callback(line)
proc.wait()
with proc_lock:
if proc in active_processes:
active_processes.remove(proc)
return proc.returncode == 0
except Exception as e:
if status_callback: status_callback(f"Error: {e}")
return False
def run_vmaf_check(reference, distorted, status_callback=None):
"""Run ab-av1 vmaf to get score"""
# Use common dependency check to find binary if needed, but here just assume it's in path or bin
ab_exe = "ab-av1"
# Check if bundled exists
bundled = Path(__file__).parent / "bin" / "ab-av1.exe"
if bundled.exists():
ab_exe = str(bundled)
cmd = [ab_exe, "vmaf", "--reference", str(reference), "--distorted", str(distorted)]
if status_callback: status_callback("Calculating VMAF...")
try:
result = subprocess.run(cmd, capture_output=True, text=True, check=True)
for line in result.stdout.splitlines():
line = line.strip()
match = re.search(r"VMAF\s+([0-9.]+)", line)
if match: return float(match.group(1))
try:
val = float(line)
if 0 <= val <= 100: return val
except: pass
return 0.0
except Exception:
return -1.0
# --- Core Logic ---
def process_file(filepath, log_category, lock_dir, log_dir, encoders, worker_id=0, status_cb=None):
"""
Process a single file.
status_cb: function(worker_id, filename, status_text, color)
"""
av1_enc, hevc_enc, hw_type = encoders
filepath = Path(filepath)
filename = filepath.name
def update(msg, color="white"):
if status_cb: status_cb(worker_id, filename, msg, color)
else: print(f"[{worker_id}] {msg}")
if shutdown_requested: return
# 1. Lock Check (Shared Storage)
lock_file = common.acquire_lock(lock_dir, filepath)
if not lock_file:
return # Locked or skipped
try:
update("Analyzing...", "blue")
# 2. Analyze Source
info = common.get_video_info(filepath)
if not info:
update("Metadata Error", "red")
return
if info["codec"] == "av1":
update("Already AV1 (Skipping)", "green")
time.sleep(1)
return
# 3. Create Samples
sample_ref = TEMP_DIR / f"{filepath.stem}_{worker_id}_ref.mkv"
sample_enc = TEMP_DIR / f"{filepath.stem}_{worker_id}_enc.mkv"
sample_start = SAMPLE_START_TIME
if info["duration"] < (SAMPLE_START_TIME + SAMPLE_DURATION):
sample_start = max(0, (info["duration"] / 2) - (SAMPLE_DURATION / 2))
update("Extracting Ref", "magenta")
cmd_ref = [
FFMPEG_BIN, "-y", "-hide_banner", "-loglevel", "error",
"-ss", str(sample_start), "-t", str(SAMPLE_DURATION),
"-i", str(filepath), "-c", "copy", "-map", "0:v:0",
str(sample_ref)
]
if not run_process(cmd_ref):
update("Extract Ref Failed", "red")
return
# TEST 1: AV1
vmaf_score = 0
savings = 0
if av1_enc:
update(f"Testing AV1 QP{DEFAULT_AV1_QP}", "yellow")
enc_args = get_encoder_args("av1", av1_enc, DEFAULT_AV1_QP)
cmd_enc = [
FFMPEG_BIN, "-y", "-hide_banner", "-loglevel", "error",
"-i", str(sample_ref), *enc_args, "-an", str(sample_enc)
]
if run_process(cmd_enc):
update("Calculating VMAF", "cyan")
vmaf_score = run_vmaf_check(sample_ref, sample_enc)
ref_size = sample_ref.stat().st_size
enc_size = sample_enc.stat().st_size
savings = (1 - (enc_size / ref_size)) * 100 if ref_size > 0 else 0
else:
update("AV1 Test Failed", "red")
chosen_codec = None
chosen_qp = 0
# Decision Logic
if vmaf_score >= TARGET_VMAF_MIN and savings >= MIN_SAVINGS_PERCENT:
chosen_codec = "av1"
chosen_qp = DEFAULT_AV1_QP
update(f"AV1 Good (VMAF {vmaf_score:.1f})", "green")
# Smart Optimization
if vmaf_score > 97.0:
update(f"Optimizing (High Quality {vmaf_score:.1f})", "yellow")
new_qp = DEFAULT_AV1_QP + 4
args_opt = get_encoder_args("av1", av1_enc, new_qp)
cmd_opt = [FFMPEG_BIN, "-y", "-hide_banner", "-loglevel", "error", "-i", str(sample_ref), *args_opt, "-an", str(sample_enc)]
if run_process(cmd_opt):
vmaf_opt = run_vmaf_check(sample_ref, sample_enc)
size_opt = sample_enc.stat().st_size
sav_opt = (1 - (size_opt / sample_ref.stat().st_size)) * 100
if vmaf_opt >= TARGET_VMAF_MIN and sav_opt > savings:
update(f"Opt Accepted (+{sav_opt - savings:.1f}%)", "green")
chosen_qp = new_qp
vmaf_score = vmaf_opt
savings = sav_opt
else:
update("Testing HEVC Fallback", "magenta")
if info["codec"] != "hevc" and hevc_enc:
hevc_args = get_encoder_args("hevc", hevc_enc, DEFAULT_HEVC_QP)
cmd_hevc = [FFMPEG_BIN, "-y", "-hide_banner", "-loglevel", "error", "-i", str(sample_ref), *hevc_args, "-an", str(sample_enc)]
run_process(cmd_hevc)
vmaf_score = run_vmaf_check(sample_ref, sample_enc)
enc_size = sample_enc.stat().st_size
savings = (1 - (enc_size / sample_ref.stat().st_size)) * 100
if vmaf_score >= TARGET_VMAF_MIN and savings >= MIN_SAVINGS_PERCENT:
update(f"HEVC Accepted (VMAF {vmaf_score:.1f})", "green")
chosen_codec = "hevc"
chosen_qp = DEFAULT_HEVC_QP
else:
update("HEVC Rejected", "red")
common.log_event(log_dir, "rejected.jsonl", {"file": str(filepath), "status": "rejected", "vmaf": vmaf_score})
else:
update("Skipping HEVC", "yellow")
# Cleanup Samples
if sample_ref.exists(): sample_ref.unlink()
if sample_enc.exists(): sample_enc.unlink()
# 4. Full Encode
if chosen_codec:
update(f"Encoding {chosen_codec.upper()} (QP {chosen_qp})", "green")
output_file = TEMP_DIR / f"{filepath.stem}.{chosen_codec}.mkv"
final_args = get_encoder_args(chosen_codec, av1_enc if chosen_codec=="av1" else hevc_enc, chosen_qp)
cmd_full = [
FFMPEG_BIN, "-y", "-hide_banner", "-loglevel", "info", "-stats",
"-i", str(filepath),
*final_args,
"-c:a", "copy", "-c:s", "copy", "-map", "0",
str(output_file)
]
def prog_cb(msg):
if "frame=" in msg:
try:
if "time=" in msg:
t_str = msg.split("time=")[1].split(" ")[0]
h, m, s = map(float, t_str.split(':'))
cur_sec = h*3600 + m*60 + s
percent = (cur_sec / info["duration"]) * 100
else:
percent = 0.0
speed = "1x"
if "speed=" in msg:
speed = msg.split("speed=")[1].split("x")[0] + "x"
update(f"Encoding {chosen_codec} | {percent:.1f}% | {speed}", "green")
except:
pass
if run_process(cmd_full, f"Full Encode ({chosen_codec.upper()} QP {chosen_qp})", status_callback=prog_cb):
final_info = common.get_video_info(output_file)
if not final_info: final_info = {"size": output_file.stat().st_size}
final_size = final_info["size"]
final_savings = (1 - (final_size / info["size"])) * 100
saved_bytes = info["size"] - final_size
update(f"Uploading (Saved {final_savings:.1f}%)", "blue")
# UPLOAD (Serialized)
with upload_lock:
update(f"Uploading...", "blue")
backup_path = filepath.with_suffix(f"{filepath.suffix}.original")
try:
shutil.move(str(filepath), str(backup_path))
shutil.copy2(str(output_file), str(filepath))
# Verify integrity
if Path(filepath).stat().st_size == final_size:
output_file.unlink()
backup_path.unlink()
# Refresh metadata for accuracy
final_info_verified = common.get_video_info(filepath)
common.log_event(log_dir, f"{log_category}.jsonl", {
"file": str(filepath),
"status": "success",
"codec": chosen_codec,
"vmaf": vmaf_score,
"savings": final_savings,
"original_metadata": info,
"encoded_metadata": final_info_verified or final_info
})
update("Done", "green")
if status_cb: status_cb(worker_id, filename, f"STATS:SAVED:{saved_bytes}", "green")
else:
update("Upload Failed (Size Mismatch)", "red")
shutil.move(str(backup_path), str(filepath))
except Exception as e:
update(f"Move Error: {str(e)[:20]}", "red")
if backup_path.exists(): shutil.move(str(backup_path), str(filepath))
else:
update("Encode Failed", "red")
if output_file.exists(): output_file.unlink()
except Exception as e:
update(f"Error: {str(e)[:30]}", "red")
finally:
if lock_file.exists(): lock_file.unlink()
update("Idle", "dim")
def main():
global debug_mode
parser = argparse.ArgumentParser()
parser.add_argument("--tv-dir", default=common.DEFAULT_CONFIG["tv_dir"])
parser.add_argument("--content-dir", default=common.DEFAULT_CONFIG["content_dir"])
parser.add_argument("--jobs", type=int, default=MAX_JOBS)
parser.add_argument("--debug", action="store_true")
parser.add_argument("--skip-until", help="Skip all files alphabetically until this filename substring is found")
parser.add_argument("--cpu-only", action="store_true", help="Force software encoding (CPU only)")
parser.add_argument("--temp-dir", help="Override local temp directory")
args = parser.parse_args()
if args.debug:
debug_mode = True
print("[Debug Mode Enabled]")
# 0. Check Dependencies
common.check_dependencies()
# 1. Setup Directories
lock_dir, log_dir = common.get_base_paths(args)
global TEMP_DIR
TEMP_DIR = common.get_temp_dir(args)
# 2. Detect Hardware
av1, hevc, hw = common.detect_hardware_encoder(args)
print("="*60)
print(f" SMART ENCODER | Hardware: {hw.upper()} | Jobs: {args.jobs}")
if hw == "cpu":
print(f" [!] Fallback to CPU Software Encoding (Slow)")
if av1: print(f" AV1: {av1} (libsvtav1)")
if hevc: print(f" HEVC: {hevc} (libx265)")
else:
print(f" AV1: {av1} | HEVC: {hevc}")
print(f" Locks: {lock_dir}")
print("="*60)
# 3. Scan & Queue
tasks = []
tv_path = Path(args.tv_dir)
if tv_path.exists():
print(f"Scanning TV: {tv_path}")
files = list(tv_path.rglob("*.mkv")) + list(tv_path.rglob("*.mp4"))
files.sort(key=lambda x: x.stat().st_size, reverse=True)
for f in files: tasks.append((f, "tv_shows"))
content_path = Path(args.content_dir)
if content_path.exists():
print(f"Scanning Content: {content_path}")
files = list(content_path.rglob("*.mkv")) + list(content_path.rglob("*.mp4"))
files.sort(key=lambda x: x.stat().st_size, reverse=True)
for f in files: tasks.append((f, "content"))
if not tasks:
print("No files found.")
return
# 4. Execute
print(f"\n🚀 Processing {len(tasks)} files...")
with ThreadPoolExecutor(max_workers=args.jobs) as executor:
futures = {
executor.submit(process_file, f, cat, lock_dir, log_dir, (av1, hevc, hw)): f
for f, cat in tasks
}
for future in as_completed(futures):
if shutdown_requested:
executor.shutdown(wait=False, cancel_futures=True)
break
try:
future.result()
except Exception as e:
print(f"Worker Error: {e}")
if __name__ == "__main__":
main()

357
src/smart_monitor.py Normal file
View File

@@ -0,0 +1,357 @@
#!/usr/bin/env python3
r"""
Smart VMAF Monitor & Dashboard
------------------------------
The main interface for the VMAF Optimizer.
Provides a TUI (Text User Interface) to visualize encoding progress.
Usage:
python smart_monitor.py --tv-dir "Z:\tv" --content-dir "Z:\content" --jobs 4 [--monitor]
"""
import os
import time
import argparse
import sys
import threading
import queue
import json
import re
from pathlib import Path
# Import the engine
import smart_gpu_encoder as encoder
import vmaf_common as common
# UI Library
try:
from rich.console import Console
from rich.live import Live
from rich.table import Table
from rich.layout import Layout
from rich.panel import Panel
from rich.text import Text
HAS_RICH = True
except ImportError:
HAS_RICH = False
print("Warning: 'rich' library not found. Running in basic mode.")
# Watchdog
try:
from watchdog.observers import Observer
from watchdog.events import FileSystemEventHandler
HAS_WATCHDOG = True
except ImportError:
HAS_WATCHDOG = False
# --- Caching ---
CACHE_FILE = Path("library_cache.json")
def load_cache():
if CACHE_FILE.exists():
try:
with open(CACHE_FILE, "r") as f:
return set(json.load(f))
except: pass
return set()
def save_cache(files):
try:
with open(CACHE_FILE, "w") as f:
# Convert paths to strings for JSON
json.dump([str(f) for f in files], f)
except: pass
def fast_scan(path):
"""Recursive scan using scandir (faster than pathlib)"""
files = []
try:
if not os.path.exists(path): return []
for entry in os.scandir(path):
if entry.is_dir():
files.extend(fast_scan(entry.path))
elif entry.is_file():
if entry.name.lower().endswith(('.mkv', '.mp4')):
if "_enc" not in entry.name and "_ref" not in entry.name:
files.append(entry.path)
except:
pass
return files
# --- UI State ---
class Dashboard:
def __init__(self, num_workers):
self.num_workers = num_workers
self.worker_status = {i: {"file": "Idle", "action": "Waiting", "progress": 0, "speed": "", "color": "dim"} for i in range(num_workers)}
self.stats = {"processed": 0, "skipped": 0, "failed": 0, "rejected": 0, "savings_gb": 0.0}
self.recent_completed = []
self.lock = threading.Lock()
def format_filename(self, filename):
# Clean Sonarr format: {Series} - S{s}E{e} - {Title} {Quality}
# Goal: Series S01E01 [Quality]
try:
# Match S01E01
match = re.search(r"(.*?) - (S\d+E\d+) - .*? ((?:Bluray|WebDL|Remux|2160p|1080p|Proper).*)\.mkv", filename, re.IGNORECASE)
if match:
series = match.group(1)[:15] # Truncate series name
s_e = match.group(2)
quality = match.group(3).split()[0] # Just take first part of quality (Bluray-2160p)
return f"{series} {s_e} [{quality}]"
# Fallback for simpler names
if len(filename) > 35:
return filename[:15] + "..." + filename[-15:]
return filename
except:
return filename
def update_worker(self, worker_id, file, action, color="blue"):
with self.lock:
display_name = self.format_filename(file)
progress = 0
speed = ""
# Parse rich status: "Encoding AV1 | 45.2% | 2.3x"
if "|" in action:
parts = action.split("|")
action_text = parts[0].strip()
for p in parts[1:]:
p = p.strip()
if "%" in p:
try: progress = float(p.replace("%", ""))
except: pass
elif "x" in p or "MB/s" in p:
speed = p
action = action_text
self.worker_status[worker_id] = {
"file": display_name,
"action": action,
"progress": progress,
"speed": speed,
"color": color
}
def add_log(self, message):
with self.lock:
ts = time.strftime("%H:%M:%S")
self.recent_completed.insert(0, f"[{ts}] {message}")
if len(self.recent_completed) > 12:
self.recent_completed.pop()
def update_stats(self, key, val=1):
with self.lock:
if key == "savings_gb": self.stats[key] += val
else: self.stats[key] += val
def get_renderable(self):
if not HAS_RICH: return ""
layout = Layout()
layout.split_column(
Layout(name="header", size=3),
Layout(name="workers", size=self.num_workers + 4),
Layout(name="stats", size=3),
Layout(name="logs", ratio=1)
)
layout["header"].update(Panel(Text("Smart GPU Encoder (AMD AMF + VMAF)", justify="center", style="bold cyan")))
# Workers Table
table = Table(box=None, expand=True, padding=(0, 1))
table.add_column("ID", width=3, style="dim", no_wrap=True)
table.add_column("File", ratio=6, no_wrap=True)
table.add_column("Action", ratio=3, no_wrap=True)
table.add_column("Progress", width=20, no_wrap=True)
table.add_column("Speed", width=12, no_wrap=True)
for i in range(self.num_workers):
ws = self.worker_status[i]
pct = ws["progress"]
# Rich Bar
if pct > 0:
bar_len = 12
filled = int(bar_len * (pct / 100))
bar_str = "━" * filled + " " * (bar_len - filled)
prog_render = Text(f"{bar_str} {pct:.1f}%", style="green")
else:
prog_render = Text("")
table.add_row(
str(i+1),
Text(ws["file"], style="white"),
Text(ws["action"], style=ws["color"]),
prog_render,
Text(ws["speed"], style="yellow")
)
layout["workers"].update(Panel(table, title="Active Workers"))
stat_str = f"Processed: [green]{self.stats['processed']}[/] | Skipped: [yellow]{self.stats['skipped']}[/] | Rejected: [magenta]{self.stats['rejected']}[/] | Failed: [red]{self.stats['failed']}[/] | Saved: [bold green]{self.stats['savings_gb']:.2f} GB[/]"
layout["stats"].update(Panel(Text.from_markup(stat_str, justify="center")))
layout["logs"].update(Panel("\n".join(self.recent_completed), title="Activity Log"))
return layout
# --- Worker Bridge ---
def worker_wrapper(worker_id, file_path, category, lock_dir, log_dir, encoders, dashboard):
def status_callback(w_id, fname, msg, color):
# Handle Stats Signal
if msg.startswith("STATS:"):
parts = msg.split(":")
if parts[1] == "SAVED":
try:
bytes_saved = float(parts[2])
dashboard.update_stats("savings_gb", bytes_saved / (1024**3))
dashboard.update_stats("processed")
except: pass
return
# Update Live Table
dashboard.update_worker(worker_id, fname, msg, color)
# Check for Completion Events to update History
# Match keywords from smart_gpu_encoder.py
if "Done" in msg:
dashboard.add_log(f"[Success] {dashboard.format_filename(fname)}")
elif "Skipped" in msg or "Skipping" in msg:
dashboard.add_log(f"[Skip] {dashboard.format_filename(fname)}")
dashboard.update_stats("skipped")
elif "Rejected" in msg:
dashboard.add_log(f"[Reject] {dashboard.format_filename(fname)}")
dashboard.update_stats("rejected")
# Rejected doesn't count as failed, just processed (or ignored stats)
elif "Failed" in msg or "Error" in msg:
dashboard.add_log(f"[Fail] {dashboard.format_filename(fname)}")
dashboard.update_stats("failed")
try:
encoder.process_file(file_path, category, lock_dir, log_dir, encoders, worker_id, status_callback)
except Exception as e:
dashboard.add_log(f"Error in worker {worker_id}: {str(e)[:30]}")
dashboard.update_stats("failed")
finally:
dashboard.update_worker(worker_id, "Idle", "Waiting", "dim")
# --- Monitor ---
class WatcherHandler(FileSystemEventHandler):
def __init__(self, queue):
self.queue = queue
def on_created(self, event):
if not event.is_directory and event.src_path.lower().endswith(('.mkv', '.mp4')):
self.queue.put(Path(event.src_path))
def on_moved(self, event):
if not event.is_directory and event.dest_path.lower().endswith(('.mkv', '.mp4')):
self.queue.put(Path(event.dest_path))
# --- Main ---
def main():
parser = argparse.ArgumentParser()
parser.add_argument("--tv-dir", default=common.DEFAULT_CONFIG["tv_dir"])
parser.add_argument("--content-dir", default=common.DEFAULT_CONFIG["content_dir"])
parser.add_argument("--jobs", type=int, default=4)
parser.add_argument("--monitor", action="store_true")
parser.add_argument("--cpu-only", action="store_true", help="Force software encoding")
parser.add_argument("--temp-dir", help="Override local temp directory")
args = parser.parse_args()
# Setup
lock_dir, log_dir = common.get_base_paths(args)
encoder.TEMP_DIR = common.get_temp_dir(args)
# Detect HW
encoders = common.detect_hardware_encoder(args)
# UI
dashboard = Dashboard(args.jobs)
dashboard.add_log(f"Logs: {log_dir}")
# Work Queue
work_queue = queue.Queue()
# Background Scanner
def background_scanner():
time.sleep(2) # Let UI start
dashboard.add_log("Starting background scan...")
# Load Cache first
cached_files = load_cache()
if cached_files:
dashboard.add_log(f"Loaded {len(cached_files)} files from cache.")
for f in cached_files:
p = Path(f)
cat = "tv_shows" if str(args.tv_dir) in str(p) else "content"
work_queue.put((p, cat))
# Real Scan
all_files = []
for d, cat in [(args.tv_dir, "tv_shows"), (args.content_dir, "content")]:
found = fast_scan(d)
for f in found:
all_files.append(f)
# Only add if NOT in cache
if str(f) not in cached_files:
work_queue.put((Path(f), cat))
dashboard.add_log(f"Scan complete. Total: {len(all_files)}")
save_cache(all_files)
# Start Scanner Thread
scan_thread = threading.Thread(target=background_scanner, daemon=True)
scan_thread.start()
# Thread Pool for Workers
threads = []
def worker_loop(w_id):
while not encoder.shutdown_requested:
try:
item = work_queue.get(timeout=1)
file_path, category = item
worker_wrapper(w_id, file_path, category, lock_dir, log_dir, encoders, dashboard)
work_queue.task_done()
except queue.Empty:
# If batch mode and scan is done and queue is empty, exit
if not args.monitor and not scan_thread.is_alive() and work_queue.empty():
time.sleep(2) # Grace period
if work_queue.empty():
return # Exit thread
continue
except Exception as e:
dashboard.add_log(f"Worker {w_id} crashed: {e}")
for i in range(args.jobs):
t = threading.Thread(target=worker_loop, args=(i,), daemon=True)
t.start()
threads.append(t)
# UI Loop
try:
if HAS_RICH:
with Live(dashboard.get_renderable(), refresh_per_second=4) as live:
while not encoder.shutdown_requested:
live.update(dashboard.get_renderable())
time.sleep(0.25)
# Exit condition
if not args.monitor and not scan_thread.is_alive() and work_queue.unfinished_tasks == 0:
# Check if threads are actually dead
if all(not t.is_alive() for t in threads):
break
else:
while not encoder.shutdown_requested:
time.sleep(1)
if not args.monitor and not scan_thread.is_alive() and work_queue.empty(): break
except KeyboardInterrupt:
encoder.shutdown_requested = True
print("\nStopping...")
print("\nDone.")
if __name__ == "__main__":
main()

47
src/test_monitor_logic.py Normal file
View File

@@ -0,0 +1,47 @@
from smart_monitor import Dashboard
import time
def test_tui_parsing():
print("Testing Dashboard Logic...")
db = Dashboard(1)
# Test 1: Encoding Progress
msg = "Encoding av1 | 45.2% | 2.3x"
db.update_worker(0, "Test File 1.mkv", msg)
status = db.worker_status[0]
print(f"Test 1 (Parsing): Progress={status['progress']} (Expected 45.2), Speed={status['speed']} (Expected 2.3x)")
assert status['progress'] == 45.2
assert status['speed'] == "2.3x"
# Test 2: Stats - Skipped
print("\nTest 2: Stats - Skipped")
# Simulate worker wrapper logic
msg = "Already AV1 (Skipping)"
if "Skipping" in msg:
db.update_stats("skipped")
print(f"Skipped Count: {db.stats['skipped']} (Expected 1)")
assert db.stats['skipped'] == 1
# Test 3: Stats - Rejected
print("\nTest 3: Stats - Rejected")
msg = "HEVC Rejected"
if "Rejected" in msg:
db.update_stats("rejected")
print(f"Rejected Count: {db.stats['rejected']} (Expected 1)")
assert db.stats['rejected'] == 1
# Test 4: Stats - Failed
print("\nTest 4: Stats - Failed")
msg = "Error: FFMPEG failed"
if "Error" in msg:
db.update_stats("failed")
print(f"Failed Count: {db.stats['failed']} (Expected 1)")
assert db.stats['failed'] == 1
print("\nAll Tests Passed!")
if __name__ == "__main__":
test_tui_parsing()

295
src/vmaf_common.py Normal file
View File

@@ -0,0 +1,295 @@
import os
import sys
import shutil
import json
import hashlib
import platform
import subprocess
import time
from pathlib import Path
from datetime import datetime
# --- Defaults ---
DEFAULT_CONFIG = {
"tv_dir": r"Z:\tv",
"content_dir": r"Z:\content",
"target_vmaf": 93.0,
"min_savings": 12.0,
"gpu_jobs": 2,
"cpu_jobs": 1,
"av1_crf": 34,
"hevc_crf": 28,
"temp_dir_windows": r"C:\Users\bnair\Videos\encodes",
"temp_dir_linux": "~/Videos/encodes"
}
# --- Paths ---
def get_base_paths(args=None):
"""
Determine root paths for locks and logs.
Priority:
1. Shared Network Drive (parent of tv_dir) -> Z:\.vmaf_locks
2. Local Fallback (current dir / logs)
"""
tv_dir = Path(args.tv_dir) if args and hasattr(args, 'tv_dir') else Path(DEFAULT_CONFIG["tv_dir"])
# Logic: Locks MUST be at the common root of content to be shared.
# We assume tv_dir is like "Z:\tv", so shared root is "Z:\"
shared_root = tv_dir.parent
# Defaults
lock_dir = Path("locks").resolve()
log_dir = Path("logs").resolve()
# Network Logic
if shared_root.exists():
network_lock = shared_root / ".vmaf_locks"
# We prefer local logs to avoid network I/O issues during logging,
# but locks MUST be shared.
try:
network_lock.mkdir(parents=True, exist_ok=True)
lock_dir = network_lock
except Exception as e:
print(f"[Warning] Could not create network locks at {network_lock}: {e}")
print(f" Falling back to local locks: {lock_dir}")
# Ensure existence
lock_dir.mkdir(parents=True, exist_ok=True)
log_dir.mkdir(parents=True, exist_ok=True)
return lock_dir, log_dir
def get_temp_dir(args=None):
# Check args override
if args and hasattr(args, 'temp_dir') and args.temp_dir:
path = Path(args.temp_dir)
elif platform.system() == "Windows":
path = Path(DEFAULT_CONFIG["temp_dir_windows"])
else:
path = Path(DEFAULT_CONFIG["temp_dir_linux"]).expanduser()
path.mkdir(parents=True, exist_ok=True)
return path
# --- Logging ---
def log_event(log_dir, filename, data):
log_path = Path(log_dir) / filename
data["timestamp"] = datetime.now().isoformat()
try:
with open(log_path, "a", encoding="utf-8") as f:
f.write(json.dumps(data) + "\n")
except Exception as e:
print(f"[ERROR] Failed to write log: {e}")
# --- Dependencies ---
def check_dependencies(required_tools=None):
if required_tools is None:
required_tools = ["ffmpeg", "ffprobe", "ab-av1"]
missing = []
# Determine bin path relative to this file
bin_path = Path(__file__).parent.parent / "bin"
for tool in required_tools:
if tool == "ab-av1":
# Check bundled first in bin/
bundled = bin_path / "ab-av1.exe"
if bundled.exists(): continue
# Check PATH
if shutil.which("ab-av1"): continue
missing.append("ab-av1")
else:
if not shutil.which(tool):
missing.append(tool)
if missing:
print(f"[!] CRITICAL: Missing tools: {', '.join(missing)}")
print(f" Checked bundled path: {bin_path}")
sys.exit(1)
# --- Metadata ---
def get_video_info(filepath):
"""Get video info using ffprobe (Standardized)"""
try:
cmd = [
"ffprobe", "-v", "quiet",
"-print_format", "json",
"-show_format", "-show_streams",
str(filepath)
]
result = subprocess.run(cmd, capture_output=True, text=True, check=True)
data = json.loads(result.stdout)
video = next((s for s in data["streams"] if s["codec_type"] == "video"), None)
if not video: return None
size = int(data["format"].get("size", 0))
duration = float(data["format"].get("duration", 0))
# Bitrate calc
bitrate = int(data["format"].get("bitrate", 0))
if bitrate == 0 and duration > 0:
bitrate = int((size * 8) / duration)
return {
"codec": video.get("codec_name"),
"width": int(video.get("width", 0)),
"height": int(video.get("height", 0)),
"duration": duration,
"size": size,
"bitrate": bitrate,
"fps": video.get("r_frame_rate", "0/0")
}
except Exception:
return None
# --- Locks ---
def acquire_lock(lock_dir, filepath):
"""
Simple file-based lock.
Returns lock_path if acquired, None if failed.
"""
# Use hash of filename to avoid long paths/invalid chars
fhash = hashlib.md5(str(filepath.name).encode()).hexdigest()
lock_file = lock_dir / f"{fhash}.lock"
if lock_file.exists():
# Check staleness (24h)
try:
if time.time() - lock_file.stat().st_mtime > 86400:
lock_file.unlink()
else:
return None
except:
return None
try:
lock_file.touch()
return lock_file
except:
return None
# --- Hardware Detection ---
def detect_hardware_encoder(args=None):
"""Detects available hardware encoders via ffmpeg (Cross-Platform)"""
# Check for forced CPU mode via args
if args and getattr(args, 'cpu_only', False):
# We still need to determine WHICH software encoder to use
# But we skip HW checks.
try:
res = subprocess.run(["ffmpeg", "-hide_banner", "-encoders"], capture_output=True, text=True)
out = res.stdout
if "libsvtav1" in out: return "libsvtav1", "libx265", "cpu"
if "libx265" in out: return None, "libx265", "cpu"
return None, None, "cpu"
except:
return None, None, "cpu"
try:
# Run ffmpeg -encoders
res = subprocess.run(["ffmpeg", "-hide_banner", "-encoders"], capture_output=True, text=True)
out = res.stdout
av1_enc = None
hevc_enc = None
hw_type = "cpu"
# 1. NVIDIA (NVENC) - Windows/Linux
if "av1_nvenc" in out: av1_enc = "av1_nvenc"
if "hevc_nvenc" in out: hevc_enc = "hevc_nvenc"
if av1_enc or hevc_enc: return av1_enc, hevc_enc, "nvenc"
# 2. AMD (AMF) - Windows
if "av1_amf" in out: av1_enc = "av1_amf"
if "hevc_amf" in out: hevc_enc = "hevc_amf"
if av1_enc or hevc_enc: return av1_enc, hevc_enc, "amf"
# 3. AMD (VAAPI) - Linux
# Often named hevc_vaapi, av1_vaapi
if "av1_vaapi" in out: av1_enc = "av1_vaapi"
if "hevc_vaapi" in out: hevc_enc = "hevc_vaapi"
if av1_enc or hevc_enc: return av1_enc, hevc_enc, "vaapi"
# 4. Intel (QSV) - Windows/Linux
if "av1_qsv" in out: av1_enc = "av1_qsv"
if "hevc_qsv" in out: hevc_enc = "hevc_qsv"
if av1_enc or hevc_enc: return av1_enc, hevc_enc, "qsv"
# 5. Apple Silicon (VideoToolbox) - macOS
if "av1_videotoolbox" in out: av1_enc = "av1_videotoolbox"
if "hevc_videotoolbox" in out: hevc_enc = "hevc_videotoolbox"
if av1_enc or hevc_enc: return av1_enc, hevc_enc, "videotoolbox"
# Fallback to Software if no HW found
# libsvtav1 / libx265
if "libsvtav1" in out: av1_enc = "libsvtav1"
if "libx265" in out: hevc_enc = "libx265"
if av1_enc or hevc_enc:
return av1_enc, hevc_enc, "cpu"
return None, None, "none"
except Exception as e:
print(f"[Warning] HW Detection failed: {e}")
return None, None, "error"
def get_encoder_args(codec, encoder, qp):
"""Returns correct ffmpeg args for specific HW vendor"""
if not encoder: return []
# Software (CPU)
if encoder == "libsvtav1":
# CRF 0-63 (Lower is better)
return ["-c:v", "libsvtav1", "-crf", str(qp), "-preset", "6", "-g", "240"]
if encoder == "libx265":
return ["-c:v", "libx265", "-crf", str(qp), "-preset", "medium"]
# NVIDIA NVENC
if "nvenc" in encoder:
# p6 = better quality, spatial-aq for better perception
return ["-c:v", encoder, "-rc", "constqp", "-qp", str(qp), "-preset", "p6", "-spatial-aq", "1"]
# AMD AMF
if "amf" in encoder:
return ["-c:v", encoder, "-usage", "transcoding", "-rc", "cqp", "-qp_i", str(qp), "-qp_p", str(qp), "-qp_b", str(qp), "-quality", "quality"]
# Intel QSV
if "qsv" in encoder:
return ["-c:v", encoder, "-global_quality", str(qp), "-look_ahead", "1"]
# Apple VideoToolbox
if "videotoolbox" in encoder:
# Map 0-51 QP to 100-0 Quality (approx)
q = int(100 - (qp * 2))
return ["-c:v", encoder, "-q:v", str(q)]
# Linux VAAPI
if "vaapi" in encoder:
# Uses -qp normally? or -global_quality? Depends on driver.
# Often needs: -vf format=nv12,hwupload
# Safe bet for vaapi is usually CQP via -qp or -global_quality
return ["-c:v", encoder, "-qp", str(qp)]
return []
def scan_directory(directory, extensions=None):
"""Scan directory for video files"""
if extensions is None:
extensions = {".mkv", ".mp4", ".mov", ".avi", ".ts"}
files = []
for dirpath, dirnames, filenames in os.walk(directory):
# Skip processed/system files
dirnames[:] = [d for d in dirnames if not d.startswith("_") and d not in [".recycle", "@eaDir"]]
for filename in filenames:
filepath = Path(dirpath) / filename
if filepath.suffix.lower() in extensions:
if "_av1" in filepath.stem.lower() or "_hevc" in filepath.stem.lower():
continue
files.append(filepath)
return sorted(files, key=lambda x: x.stat().st_size, reverse=True)

42
test_run.py Normal file
View File

@@ -0,0 +1,42 @@
import os
import subprocess
from pathlib import Path
# Setup
TEST_DIR = Path("test_media")
TEST_DIR.mkdir(exist_ok=True)
TEST_FILE = TEST_DIR / "test_video.mkv"
print("--- SMART ENCODER TEST ---")
# 1. Create Dummy Video (10s, 1080p, noise to ensure it's not trivial)
if not TEST_FILE.exists():
print("Generating dummy video...")
# Generate 10s video with noise
cmd = [
"ffmpeg", "-y", "-f", "lavfi", "-i", "testsrc=duration=10:size=1920x1080:rate=30",
"-f", "lavfi", "-i", "sine=frequency=1000:duration=10",
"-c:v", "libx264", "-pix_fmt", "yuv420p", "-b:v", "5M",
"-c:a", "aac",
str(TEST_FILE)
]
subprocess.run(cmd, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
print(f"Created {TEST_FILE}")
# 2. Run Encoder in Debug Mode
print("\nRunning Smart Encoder on test file...")
print("(This should show detailed ffmpeg output)")
print("="*60)
cmd = [
"python", "smart_gpu_encoder.py",
"--tv-dir", str(TEST_DIR.resolve()),
"--content-dir", str(TEST_DIR.resolve()),
"--jobs", "1",
"--debug"
]
subprocess.run(cmd)
print("\n--- TEST COMPLETE ---")
print("Check for .mkv files in C:\\Users\\bnair\\Videos\\encodes")