Compare commits
8 Commits
91418fa898
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
b1d7630a10 | ||
|
|
ef81365fce | ||
|
|
d5bcaf16ee | ||
|
|
51fc7e12bc | ||
|
|
4729c75e41 | ||
|
|
f1e79ad01d | ||
|
|
b4a82e0db5 | ||
|
|
05a306dc42 |
350
FEATURES.md
Normal file
350
FEATURES.md
Normal file
@@ -0,0 +1,350 @@
|
||||
# optimize_library.py - Complete Feature Restore
|
||||
|
||||
## Overview
|
||||
|
||||
Restored all degraded functionality from original `optimise_media_v2.py` and added new features:
|
||||
- Intelligent VMAF targeting (94 → 93 → estimate for 12% savings)
|
||||
- Comprehensive logging system (separate logs for tv/movies vs content)
|
||||
- Before/after metadata tracking
|
||||
- Hardware encoding with 1 HW worker + CPU workers
|
||||
- Plex refresh on completion
|
||||
- Resume capability and graceful shutdown
|
||||
- Lock file coordination for multi-machine setups
|
||||
|
||||
---
|
||||
|
||||
## Key Features
|
||||
|
||||
### 1. Intelligent VMAF Target Search
|
||||
|
||||
**Flow:**
|
||||
```
|
||||
Try VMAF 94
|
||||
↓ Success? → Check savings ≥ 12%
|
||||
↓ Yes ↓ No
|
||||
Encode at 94 Try VMAF 93
|
||||
↓
|
||||
Savings ≥ 12%?
|
||||
↓ Yes ↓ No
|
||||
Encode at 93 Find 15% (test 92, 90)
|
||||
```
|
||||
|
||||
**Benefits:**
|
||||
- Targets high quality first (VMAF 94)
|
||||
- Falls back to VMAF 93 if needed
|
||||
- Estimates VMAF for 15%+ savings if both fail
|
||||
- Logs recommendations for manual review
|
||||
|
||||
### 2. Comprehensive Logging System
|
||||
|
||||
**Log Files (in log_dir):**
|
||||
- `tv_movies.jsonl` - Successful encodes from /tv and /movies
|
||||
- `content.jsonl` - Successful encodes from /content
|
||||
- `failed_encodes.jsonl` - Encoding errors
|
||||
- `failed_searches.jsonl` - Files that couldn't hit any VMAF target
|
||||
- `low_savings_skips.jsonl` - Files with <12% savings + 15% estimates
|
||||
|
||||
**Log Entry Structure:**
|
||||
```json
|
||||
{
|
||||
"file": "/path/to/file.mkv",
|
||||
"status": "success",
|
||||
"vmaf": 94.0,
|
||||
"crf": 37.0,
|
||||
"before": {
|
||||
"codec": "h264",
|
||||
"width": 1280,
|
||||
"height": 720,
|
||||
"bitrate": 1010,
|
||||
"size": 158176376,
|
||||
"duration": 1252.298
|
||||
},
|
||||
"after": {
|
||||
"codec": "av1",
|
||||
"width": 1280,
|
||||
"height": 720,
|
||||
"bitrate": 775,
|
||||
"size": 121418115,
|
||||
"duration": 1252.296
|
||||
},
|
||||
"duration": 1299.28,
|
||||
"savings": 23.24,
|
||||
"timestamp": "2025-12-31T13:56:55.894288"
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Before/After Metadata
|
||||
|
||||
**Tracked metrics:**
|
||||
- Codec (h264, hevc, av1, etc.)
|
||||
- Resolution (width × height)
|
||||
- Bitrate (calculated from size × 8 / duration - more reliable than ffprobe)
|
||||
- File size (bytes)
|
||||
- Duration (seconds)
|
||||
- Savings percentage
|
||||
|
||||
**Why calculate bitrate from file size?**
|
||||
- FFmpeg's bitrate field often returns 0 for VBR files
|
||||
- File size + duration = accurate, reliable metric
|
||||
|
||||
### 4. Hardware Encoding with 1 HW Worker
|
||||
|
||||
**Configuration:**
|
||||
```bash
|
||||
# Enable hardware encoding with 1 HW worker + rest CPU
|
||||
python3 optimize_library.py /media --hwaccel auto --use-hardware-worker --workers 4
|
||||
```
|
||||
|
||||
**Behavior:**
|
||||
- First file processed: Uses hardware encoding (faster, GPU-accelerated)
|
||||
- Remaining files: Use CPU encoding (slower, more accurate)
|
||||
- Hardware methods auto-detected:
|
||||
- Windows: d3d11va
|
||||
- macOS: videotoolbox
|
||||
- Linux/WSL: vaapi
|
||||
|
||||
**Why 1 HW worker?**
|
||||
- GPU memory is limited - multiple simultaneous encodes may OOM
|
||||
- CPU encoding yields higher quality at same CRF
|
||||
- Best of both worlds: 1 fast GPU encode, rest high-quality CPU encodes
|
||||
|
||||
**To disable hardware:**
|
||||
```bash
|
||||
python3 optimize_library.py /media --hwaccel none
|
||||
# or just omit --hwaccel flag
|
||||
```
|
||||
|
||||
### 5. Separate Logging by Directory
|
||||
|
||||
**Automatic detection:**
|
||||
- Scanning `/mnt/Media/tv` or `/mnt/Media/movies` → Logs to `tv_movies.jsonl`
|
||||
- Scanning `/mnt/Media/content` → Logs to `content.jsonl`
|
||||
|
||||
**Exclusion:**
|
||||
- When scanning `/tv` or `/movies`, the `/content` subdirectory is automatically excluded
|
||||
|
||||
**Example:**
|
||||
```bash
|
||||
# TV/Movies - logged together
|
||||
python3 optimize_library.py /mnt/Media/movies
|
||||
# Creates: tv_movies.jsonl
|
||||
|
||||
# Content - logged separately
|
||||
python3 optimize_library.py /mnt/Media/content
|
||||
# Creates: content.jsonl
|
||||
```
|
||||
|
||||
### 6. Plex Refresh on Completion
|
||||
|
||||
**Configuration:**
|
||||
```bash
|
||||
python3 optimize_library.py /media \
|
||||
--plex-url http://localhost:32400 \
|
||||
--plex-token YOUR_TOKEN_HERE
|
||||
```
|
||||
|
||||
**Behavior:**
|
||||
- After all files processed (or shutdown), triggers Plex library refresh
|
||||
- Only refreshes if at least 1 file was successfully encoded
|
||||
- Uses Plex API: `GET /library/sections/1/refresh`
|
||||
|
||||
**To get Plex token:**
|
||||
1. Sign in to Plex Web
|
||||
2. Go to Settings → Network
|
||||
3. Look for "List of IP addresses and ports that have authorized devices"
|
||||
4. Copy token (long alphanumeric string)
|
||||
|
||||
### 7. Resume Capability
|
||||
|
||||
**Automatic skip:**
|
||||
- Files already processed in current run are skipped
|
||||
- Uses lock files for multi-machine coordination
|
||||
- Press Ctrl+C for graceful shutdown
|
||||
|
||||
**Lock file mechanism:**
|
||||
```
|
||||
/log_dir/.lock/{video_filename}
|
||||
```
|
||||
|
||||
- Before processing: Check if lock exists → Skip if yes
|
||||
- Start processing: Create lock
|
||||
- Finish processing: Remove lock
|
||||
|
||||
**Multi-machine safe:**
|
||||
- Machine A: No lock → Create lock → Encode → Remove lock
|
||||
- Machine B: Lock exists → Skip file
|
||||
- Result: Different machines process different files automatically
|
||||
|
||||
### 8. Graceful Shutdown
|
||||
|
||||
**Behavior:**
|
||||
- Press Ctrl+C → Current tasks finish, new tasks stop
|
||||
- Output: "⚠️ Shutdown requested. Finishing current tasks..."
|
||||
- No partial encodes left hanging
|
||||
|
||||
---
|
||||
|
||||
## Usage Examples
|
||||
|
||||
### Basic Usage (CPU only)
|
||||
```bash
|
||||
python3 optimize_library.py /mnt/Media/movies
|
||||
```
|
||||
|
||||
### Hardware Encoding (1 HW + 3 CPU workers)
|
||||
```bash
|
||||
python3 optimize_library.py /mnt/Media/movies \
|
||||
--hwaccel auto \
|
||||
--use-hardware-worker \
|
||||
--workers 4
|
||||
```
|
||||
|
||||
### With Plex Refresh
|
||||
```bash
|
||||
python3 optimize_library.py /mnt/Media/tv \
|
||||
--plex-url http://localhost:32400 \
|
||||
--plex-token YOUR_TOKEN \
|
||||
--workers 2
|
||||
```
|
||||
|
||||
### Custom Settings
|
||||
```bash
|
||||
python3 optimize_library.py /mnt/Media/movies \
|
||||
--vmaf 95 \
|
||||
--preset 7 \
|
||||
--workers 3 \
|
||||
--log-dir /custom/logs \
|
||||
--hwaccel vaapi
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## New Command-Line Arguments
|
||||
|
||||
| Argument | Description | Default |
|
||||
|----------|-------------|----------|
|
||||
| `directory` | Root directory to scan | (required) |
|
||||
| `--vmaf` | Target VMAF score | 95.0 |
|
||||
| `--preset` | SVT-AV1 Preset (4=best, 8=fast) | 6 |
|
||||
| `--workers` | Concurrent files to process | 1 |
|
||||
| `--samples` | Samples for CRF search | 4 |
|
||||
| `--hwaccel` | Hardware acceleration (auto, vaapi, d3d11va, videotoolbox, none) | None |
|
||||
| `--use-hardware-worker` | Use 1 hardware worker + rest CPU (requires --hwaccel) | False |
|
||||
| `--plex-url` | Plex server URL | None |
|
||||
| `--plex-token` | Plex authentication token | None |
|
||||
| `--log-dir` | Log directory | /opt/Optmiser/logs |
|
||||
|
||||
---
|
||||
|
||||
## Monitoring and Diagnostics
|
||||
|
||||
### Check logs in real-time
|
||||
```bash
|
||||
# Watch successful encodes
|
||||
tail -f /opt/Optmiser/logs/tv_movies.jsonl | jq '.'
|
||||
|
||||
# Check files logged for review (low savings)
|
||||
tail -f /opt/Optmiser/logs/low_savings_skips.jsonl | jq '.'
|
||||
```
|
||||
|
||||
### View statistics
|
||||
```bash
|
||||
# Count successful encodes
|
||||
jq -r '.status' /opt/Optmiser/logs/tv_movies.jsonl | sort | uniq -c
|
||||
|
||||
# Average savings
|
||||
jq -r '.savings' /opt/Optmiser/logs/tv_movies.jsonl | jq -s 'add/length'
|
||||
|
||||
# Files by VMAF target
|
||||
jq -r '.vmaf' /opt/Optmiser/logs/tv_movies.jsonl | sort | uniq -c
|
||||
```
|
||||
|
||||
### Check lock files (multi-machine coordination)
|
||||
```bash
|
||||
ls -la /opt/Optmiser/.lock/
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Hardware encoding not working
|
||||
```bash
|
||||
# Check if hwaccel is detected
|
||||
python3 optimize_library.py /media --hwaccel auto --help
|
||||
|
||||
# Verify ffmpeg supports hardware acceleration
|
||||
ffmpeg -hwaccels
|
||||
|
||||
# Try specific hardware method
|
||||
python3 optimize_library.py /media --hwaccel vaapi
|
||||
```
|
||||
|
||||
### Plex refresh not working
|
||||
```bash
|
||||
# Test curl manually
|
||||
curl -X GET http://localhost:32400/library/sections/1/refresh \
|
||||
-H "X-Plex-Token: YOUR_TOKEN"
|
||||
|
||||
# Check Plex token is valid
|
||||
curl -X GET http://localhost:32400/library/sections \
|
||||
-H "X-Plex-Token: YOUR_TOKEN"
|
||||
```
|
||||
|
||||
### Files being skipped (locked)
|
||||
```bash
|
||||
# Check for stale lock files
|
||||
ls -la /opt/Optmiser/.lock/
|
||||
|
||||
# Remove stale locks (if no process is running)
|
||||
rm /opt/Optmiser/.lock/*.lock
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Differences from Original optimise_media_v2.py
|
||||
|
||||
### Preserved (restored):
|
||||
- ✅ Intelligent VMAF target search (94 → 93 → 15% estimate)
|
||||
- ✅ Comprehensive logging system (5 log files)
|
||||
- ✅ Before/after metadata (codec, bitrate, size, duration)
|
||||
- ✅ Real-time output streaming
|
||||
- ✅ Lock file mechanism for multi-machine
|
||||
- ✅ Recommendations system
|
||||
|
||||
### Added new features:
|
||||
- ✅ Hardware encoding with 1 HW worker + CPU workers
|
||||
- ✅ Separate logging for /tv+/movies vs /content
|
||||
- ✅ Plex refresh on completion
|
||||
- ✅ Graceful shutdown (Ctrl+C handling)
|
||||
- ✅ Resume capability (track processed files)
|
||||
- ✅ Configurable log directory
|
||||
|
||||
### Changed from original:
|
||||
- `AB_AV1_PATH` → Use system `ab-av1` (more portable)
|
||||
- Fixed `--enc-input` usage (only for crf-search, not encode)
|
||||
- Added proper exception handling for probe failures
|
||||
- Improved error messages and progress output
|
||||
|
||||
---
|
||||
|
||||
## Performance Characteristics
|
||||
|
||||
### Single file encoding time (1-hour 1080p h264)
|
||||
| Method | VMAF 94 | VMAF 93 | VMAF 90 |
|
||||
|--------|------------|------------|------------|
|
||||
| CPU (24 threads) | 4-5 min | 3-4 min | 2-3 min |
|
||||
| GPU (hardware) | 30-60 sec | 20-40 sec | 15-30 sec |
|
||||
|
||||
### Multi-worker throughput
|
||||
| Workers | HW worker | Throughput (1-hour files) |
|
||||
|---------|-------------|---------------------------|
|
||||
| 1 | No | ~1 file per 5 min (CPU) |
|
||||
| 1 | Yes | ~1 file per 1 min (GPU) |
|
||||
| 4 | No | ~4 files per 5 min (CPU) |
|
||||
| 4 | Yes | ~1 GPU file + 3 CPU files (~4 total) |
|
||||
|
||||
---
|
||||
|
||||
**Last Updated:** December 31, 2025
|
||||
**Version:** 3.0 - Complete Feature Restore
|
||||
52
MAINTENANCE.md
Normal file
52
MAINTENANCE.md
Normal file
@@ -0,0 +1,52 @@
|
||||
# VMAF Optimizer - Maintenance Guide
|
||||
|
||||
## Project Goals
|
||||
To automatically optimize a large video library (TV Shows & Content) on Windows using an **AMD RX 9070 XT** GPU.
|
||||
The goal is to reduce file size while maintaining "Visually Lossless" quality (VMAF 93+).
|
||||
|
||||
## Core Strategy
|
||||
1. **Hybrid Encoding:**
|
||||
* **Encode:** Use Hardware Acceleration (`av1_amf` / `hevc_amf`) for speed (critical for 2TB+ library).
|
||||
* **Verify:** Use `ab-av1` (software) strictly for VMAF score calculation.
|
||||
2. **Safety Net:**
|
||||
* Extract a 60s sample.
|
||||
* Encode sample with AV1 (QP 32).
|
||||
* Calculate VMAF & Savings.
|
||||
* **Decision Logic:**
|
||||
* If VMAF > 97: Re-encode with higher QP (more compression) to save space.
|
||||
* If VMAF < 93 or Savings < 12%: Fallback to HEVC.
|
||||
* If HEVC fails: Keep original.
|
||||
3. **Architecture:**
|
||||
* `smart_gpu_encoder.py`: The "Engine". Handles logic, ffmpeg calls, VMAF checks. Can run standalone.
|
||||
* `smart_monitor.py`: The "UI/Controller". Runs the engine in threads, displays a TUI (Rich), handles caching and directory scanning.
|
||||
|
||||
## Current Status
|
||||
* **Working:**
|
||||
* Hardware detection (AMD/NVIDIA/Intel).
|
||||
* VMAF calculation (fixed regex for raw number output).
|
||||
* Optimization logic (Smart QP adjustment).
|
||||
* Multithreading (4 workers).
|
||||
* Caching (Instant startup on second run).
|
||||
* **Issues:**
|
||||
* **UI Progress:** The TUI shows "Action" but Progress/Speed columns are empty.
|
||||
* *Root Cause:* `smart_gpu_encoder.py` sends status updates like `Encoding AV1 (size=...)` but `smart_monitor.py` expects format `Action | Progress% | Speed`.
|
||||
* **Logs:** User reported logs might be hard to find or missing if permissions fail on `Z:`.
|
||||
|
||||
## Debugging Instructions
|
||||
To run a simplified test without scanning the whole library:
|
||||
1. Create a folder `test_media` with 1 small video file.
|
||||
2. Run:
|
||||
```powershell
|
||||
python smart_gpu_encoder.py --tv-dir ".\test_media" --content-dir ".\test_media" --jobs 1
|
||||
```
|
||||
|
||||
## Future Todos
|
||||
1. **Fix TUI Parsing:** Update `smart_gpu_encoder.py` to format strings as `Action | Percent% | Speed`.
|
||||
2. **Sonarr Integration:** Add webhook triggers on success.
|
||||
3. **Linux Support:** Verify `av1_qsv` or `av1_vaapi` flags for Linux/Docker deployments.
|
||||
|
||||
---
|
||||
**Key Files:**
|
||||
* `smart_gpu_encoder.py`: Core logic.
|
||||
* `smart_monitor.py`: TUI and Watchdog.
|
||||
* `library_cache.json`: Cache of scanned files to speed up startup.
|
||||
649
README.md
649
README.md
@@ -1,556 +1,103 @@
|
||||
# VMAF Optimiser
|
||||
|
||||
Automated video library optimization to AV1 using VMAF (Video Multimethod Assessment Fusion) quality targets. Intelligently searches for optimal encoding parameters and gracefully degrades quality when needed to achieve target file size savings.
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Requirements
|
||||
|
||||
**Prerequisites:**
|
||||
- Python 3.8+
|
||||
- FFmpeg with VMAF support (`ffmpeg -filters 2>&1 | grep libvmaf`)
|
||||
- ab-av1 binary (v0.10.3+)
|
||||
|
||||
**Installation:**
|
||||
```bash
|
||||
# Install ab-av1 via cargo
|
||||
cargo install ab-av1
|
||||
|
||||
# Or download pre-built binary
|
||||
wget https://github.com/alexheretic/ab-av1/releases/download/v0.10.3/ab-av1-x86_64-unknown-linux-musl
|
||||
chmod +x ab-av1
|
||||
```
|
||||
|
||||
## Running the Optimiser
|
||||
|
||||
### Windows / macOS
|
||||
|
||||
Use the PowerShell wrapper script:
|
||||
|
||||
```powershell
|
||||
# Interactive mode (shows prompts)
|
||||
.\run_optimisation.ps1
|
||||
|
||||
# Direct execution with parameters
|
||||
.\run_optimisation.ps1 --directory "D:\Movies" --vmaf 95 --preset 6 --workers 1
|
||||
```
|
||||
|
||||
**Available flags:**
|
||||
- `--directory <path>` - Root directory to scan (default: current directory)
|
||||
- `--vmaf <score>` - Target VMAF score (default: 95.0)
|
||||
- `--preset <value>` - SVT-AV1 Preset (default: 6)
|
||||
- `--workers <count>` - Concurrent files to process (default: 1)
|
||||
- `--samples <count>` - Samples for CRF search (default: 4)
|
||||
- `--thorough` - Use thorough mode (slower, more accurate)
|
||||
- `--encoder <name>` - ab-av1 encoder (default: svt-av1)
|
||||
- `--hwaccel <value>` - Hardware acceleration (default: none)
|
||||
|
||||
### Linux / WSL
|
||||
|
||||
Use the bash wrapper script:
|
||||
|
||||
```bash
|
||||
# Interactive mode
|
||||
./run_optimisation.sh
|
||||
|
||||
# Direct execution with parameters
|
||||
./run_optimisation.sh --directory /mnt/Media/Movies --vmaf 95 --workers 1
|
||||
```
|
||||
|
||||
**Same flags as PowerShell version:**
|
||||
- `--directory <path>` - Root directory to scan
|
||||
- `--vmaf <score>` - Target VMAF score
|
||||
- `--preset <value>` - SVT-AV1 Preset
|
||||
- `--workers <count>` - Concurrent files to process
|
||||
- `--samples <count>` - Samples for CRF search
|
||||
- `--thorough` - Use thorough mode
|
||||
- `--encoder <name>` - ab-av1 encoder
|
||||
- `--hwaccel <value>` - Hardware acceleration
|
||||
|
||||
## How It Works
|
||||
|
||||
### Phase 1: Video Analysis
|
||||
1. Scans directory for video files (.mkv, .mp4)
|
||||
2. Uses `ffprobe` to get:
|
||||
- Codec (h264, hevc, etc.)
|
||||
- Resolution (width × height)
|
||||
- Bitrate (calculated from size/duration)
|
||||
- HDR status (color transfer detection)
|
||||
3. Skips if already AV1 encoded
|
||||
|
||||
### Phase 2: VMAF Target Search (Intelligent Fallback)
|
||||
|
||||
The script tries VMAF targets in **descending order** (highest quality first):
|
||||
|
||||
```
|
||||
Try VMAF 94 (Premium)
|
||||
↓
|
||||
Can achieve?
|
||||
↓ Yes ↓ No
|
||||
Calculate savings Try VMAF 93
|
||||
↓
|
||||
Savings ≥ 12%?
|
||||
↓ Yes ↓ No
|
||||
Encode at VMAF 94 Calculate savings
|
||||
↓
|
||||
Savings ≥ 12%?
|
||||
↓ Yes ↓ No
|
||||
Encode at VMAF 93 Find 15% (test 92, 90)
|
||||
```
|
||||
|
||||
**Fallback Logic:**
|
||||
- If VMAF 94 gives ≥12% savings → **Encode at VMAF 94**
|
||||
- If VMAF 94 <12% but VMAF 93 ≥12% → **Encode at VMAF 93**
|
||||
- If both <12% → Find what VMAF gives 15%+ savings:
|
||||
- Tests VMAF 93, 92, 90
|
||||
- Reports "FOUND 15%+ SAVINGS" with exact parameters
|
||||
- Logs for manual review (no encoding)
|
||||
|
||||
### Phase 3: CRF Search
|
||||
|
||||
Uses `ab-av1 crf-search` with `--thorough` flag:
|
||||
- Takes multiple samples (20-30s segments) from video
|
||||
- Interpolates binary search for optimal CRF
|
||||
- Outputs: Best CRF, Mean VMAF, Predicted size
|
||||
- Uses `--samples` to control accuracy (default: 4 samples)
|
||||
|
||||
### Phase 4: Full Encoding (with Real-time Output)
|
||||
|
||||
If savings threshold met:
|
||||
1. Runs `ab-av1 encode` with found CRF
|
||||
2. **Streams all output in real-time** (you see progress live)
|
||||
3. Shows ETA, encoding speed, frame count
|
||||
4. Uses `--acodec copy` to preserve audio/subtitles
|
||||
|
||||
**Real-time output example:**
|
||||
```
|
||||
→ Running encoding (CRF 34)
|
||||
Encoded 4320/125400 frames (3.4%)
|
||||
Encoded 8640/125400 frames (6.9%)
|
||||
Encoded 12960/125400 frames (10.3%)
|
||||
...
|
||||
Encoded 125400/125400 frames (100.0%)
|
||||
Speed: 15.2 fps, ETA: 2s
|
||||
```
|
||||
|
||||
### Phase 5: Verification & Replacement
|
||||
|
||||
1. Probes encoded file for actual stats
|
||||
2. Calculates actual savings
|
||||
3. Only replaces original if new file is smaller
|
||||
4. Converts .mp4 to .mkv if needed
|
||||
|
||||
## Configuration
|
||||
|
||||
Key settings (edit in `optimize_library.py`):
|
||||
|
||||
```python
|
||||
TARGETS = [94.0, 93.0, 92.0, 90.0] # VMAF targets to try
|
||||
MIN_SAVINGS_PERCENT = 12.0 # Encode if savings ≥12%
|
||||
TARGET_SAVINGS_FOR_ESTIMATE = 15.0 # Estimate for this level
|
||||
PRESET = 6 # SVT-AV1 preset (4=best, 8=fast)
|
||||
EXTENSIONS = {".mkv", ".mp4", ".mov", ".avi", ".ts"}
|
||||
```
|
||||
|
||||
### What is CRF?
|
||||
|
||||
**Constant Rate Factor (CRF):** Quality/bitrate trade-off
|
||||
- **Lower CRF** = Higher quality, larger files (e.g., CRF 20)
|
||||
- **Higher CRF** = Lower quality, smaller files (e.g., CRF 40)
|
||||
- **AV1 CRF range:** 0-63 (default for VMAF 94 is ~34-36)
|
||||
|
||||
### What is VMAF?
|
||||
|
||||
**Video Multimethod Assessment Fusion:** Netflix's quality metric
|
||||
- **VMAF 95:** "Visually lossless" - indistinguishable from source
|
||||
- **VMAF 94:** Premium quality - minor artifacts
|
||||
- **VMAF 93:** Good quality - acceptable for most content
|
||||
- **VMAF 90:** Standard quality - may have noticeable artifacts
|
||||
- **VMAF 85:** Acceptable quality for mobile/low bandwidth
|
||||
|
||||
## Hardware Acceleration
|
||||
|
||||
**Automatic hwaccel detection:**
|
||||
When `--hwaccel auto` is specified, the script selects appropriate hardware acceleration:
|
||||
|
||||
| Platform | Auto Selection | Notes |
|
||||
|-----------|----------------|--------|
|
||||
| Windows | d3d11va | Direct3D Video Acceleration |
|
||||
| macOS | videotoolbox | VideoToolbox framework |
|
||||
| Linux/WSL | vaapi | Video Acceleration via VA-API |
|
||||
|
||||
**Discrete GPU vs iGPU priority:**
|
||||
- **Discrete GPU (e.g., AMD RX 7900 XT) takes priority over iGPU**
|
||||
- FFmpeg/ab-av1 will prefer the more capable encoder
|
||||
- For AV1 encoding, discrete GPU is selected if present
|
||||
|
||||
**To disable hardware acceleration:**
|
||||
```powershell
|
||||
.\run_optimisation.ps1 --hwaccel none
|
||||
```
|
||||
|
||||
```bash
|
||||
./run_optimisation.sh --hwaccel none
|
||||
```
|
||||
|
||||
## Running on Multiple Machines
|
||||
|
||||
### Lock File Mechanism
|
||||
|
||||
Each video file has a corresponding lock file:
|
||||
```
|
||||
/opt/Optmiser/.lock/{video_filename}
|
||||
```
|
||||
|
||||
**Process:**
|
||||
1. Machine A checks for lock → None found, creates lock
|
||||
2. Machine A starts encoding
|
||||
3. Machine B checks for lock → Found, skips file
|
||||
4. Machine A finishes, removes lock
|
||||
5. Machine B can now process that file
|
||||
|
||||
**Result:** Different machines automatically process different files!
|
||||
|
||||
### Multi-Machine Setup
|
||||
|
||||
**Machine 1 (Linux Server - Intel i9-12900H):**
|
||||
```bash
|
||||
cd /opt/Optmiser
|
||||
git pull origin main
|
||||
./run_optimisation.sh /mnt/Media/movies --vmaf 95
|
||||
```
|
||||
|
||||
**Machine 2 (Windows PC - AMD RX 7900 XT):**
|
||||
```powershell
|
||||
cd C:\Optmiser
|
||||
git pull origin main
|
||||
.\run_optimisation.ps1 D:\Media\movies --vmaf 95 --hwaccel auto
|
||||
```
|
||||
|
||||
**Machine 3 (Another Linux PC):**
|
||||
```bash
|
||||
cd /opt/Optmiser
|
||||
git pull origin main
|
||||
./run_optimisation.sh /home/user/Media/tv --vmaf 95
|
||||
```
|
||||
|
||||
All three can run simultaneously - lock files prevent duplicates!
|
||||
|
||||
## Logging System
|
||||
|
||||
All logs stored in `/opt/Optmiser/logs/` directory:
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `tv_movies.jsonl` | Successful TV & Movie encodes |
|
||||
| `content.jsonl` | Successful Content folder encodes |
|
||||
| `low_savings_skips.jsonl` | Files with <12% savings + 15% estimates |
|
||||
| `failed_searches.jsonl` | Files that couldn't hit any VMAF target |
|
||||
| `failed_encodes.jsonl` | Encoding errors |
|
||||
|
||||
### Log Entry Format
|
||||
|
||||
**Successful encode:**
|
||||
```json
|
||||
{
|
||||
"file": "/path/to/file.mkv",
|
||||
"status": "success",
|
||||
"vmaf": 94.0,
|
||||
"crf": 34.0,
|
||||
"before": {
|
||||
"codec": "h264",
|
||||
"bitrate": 8500,
|
||||
"size": 2684354560,
|
||||
"duration": 1379.44
|
||||
},
|
||||
"after": {
|
||||
"codec": "av1",
|
||||
"bitrate": 6400,
|
||||
"size": 2013265920,
|
||||
"duration": 1379.44
|
||||
},
|
||||
"duration": 145.2,
|
||||
"savings": 25.0,
|
||||
"timestamp": "2025-12-31T12:00:00.000Z"
|
||||
}
|
||||
```
|
||||
|
||||
**Low savings with 15% estimate:**
|
||||
```json
|
||||
{
|
||||
"file": "/path/to/file.mkv",
|
||||
"vmaf_94": 94.0,
|
||||
"savings_94": 7.0,
|
||||
"vmaf_93": 93.0,
|
||||
"savings_93": 18.0,
|
||||
"target_for_15_percent": {
|
||||
"target_vmaf": 93,
|
||||
"crf": 37,
|
||||
"savings": 18.0,
|
||||
"quality_drop": 1,
|
||||
"found": true
|
||||
},
|
||||
"recommendations": "logged_for_review",
|
||||
"timestamp": "2025-12-31T12:00:00.000Z"
|
||||
}
|
||||
```
|
||||
|
||||
### Viewing Logs
|
||||
|
||||
```bash
|
||||
# Watch logs in real-time
|
||||
tail -f /opt/Optmiser/logs/tv_movies.jsonl | jq '.'
|
||||
|
||||
# Check files logged for review (both 94 and 93 <12%)
|
||||
cat /opt/Optmiser/logs/low_savings_skips.jsonl | jq '.[] | select(.recommendations=="logged_for_review")'
|
||||
|
||||
# Statistics
|
||||
jq -r '.status' /opt/Optmiser/logs/tv_movies.jsonl | sort | uniq -c
|
||||
|
||||
# Find what CRF/VMAF combinations are being used most
|
||||
jq -r '[.vmaf, .crf] | @tsv' /opt/Optmiser/logs/tv_movies.jsonl | sort | uniq -c
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Issue: "0k bitrate" display
|
||||
|
||||
**Cause:** VBR (Variable Bitrate) files show 0 in ffprobe's format bitrate field.
|
||||
|
||||
**Solution:** Calculate from `(size × 8) / duration`
|
||||
|
||||
### Issue: Multiple machines encoding same file
|
||||
|
||||
**Cause:** No coordination between machines.
|
||||
|
||||
**Solution:** Lock files in `/opt/Optmiser/.lock/{video_filename}`
|
||||
|
||||
### Issue: Encode fails with "unexpected argument"
|
||||
|
||||
**Cause:** Using wrong flags for ab-av1 commands.
|
||||
|
||||
**Solution:** Script now validates ab-av1 support at runtime and warns gracefully.
|
||||
|
||||
### Issue: Out of Memory
|
||||
|
||||
**Solution:** Reduce workers or increase swap:
|
||||
```bash
|
||||
# Increase swap (if needed)
|
||||
sudo fallocate -l 4G /swapfile
|
||||
sudo chmod 600 /swapfile
|
||||
sudo mkswap /swapfile
|
||||
sudo swapon /swapfile
|
||||
|
||||
# Use more conservative settings
|
||||
./run_optimisation.sh --workers 1 --vmaf 93
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
### For Servers (Intel i9-12900H)
|
||||
|
||||
1. **Use 50% CPU mode** if running other services (Plex, Jellyfin):
|
||||
```bash
|
||||
./run_optimisation.sh --workers 1 --cpu-limit 50
|
||||
```
|
||||
|
||||
2. **Run during off-peak hours** to minimize impact on users
|
||||
|
||||
3. **Monitor CPU temperature**:
|
||||
```bash
|
||||
watch -n 2 'sensors | grep "Package id"'
|
||||
```
|
||||
|
||||
4. **Use higher preset for faster encodes** (preset 7-8):
|
||||
```bash
|
||||
./run_optimisation.sh --preset 8 --vmaf 93
|
||||
```
|
||||
|
||||
### For Windows PC (AMD RX 7900 XT)
|
||||
|
||||
1. **Enable hardware acceleration** for massive speedup:
|
||||
```powershell
|
||||
.\run_optimisation.ps1 --hwaccel auto
|
||||
```
|
||||
|
||||
2. **Test small sample first** to verify settings:
|
||||
```powershell
|
||||
.\run_optimisation.ps1 --directory "D:\Media\sample" --thorough --vmaf 95
|
||||
```
|
||||
|
||||
3. **Monitor GPU usage**:
|
||||
```powershell
|
||||
# Task Manager or radeontop (if available)
|
||||
```
|
||||
|
||||
4. **Consider quality compensation:** GPU encoding may need slightly lower VMAF target (e.g., VMAF 92) to match CPU quality.
|
||||
|
||||
### For WSL
|
||||
|
||||
1. **Access Windows drives via /mnt/c/:**
|
||||
```bash
|
||||
ls /mnt/c/Media/movies
|
||||
```
|
||||
|
||||
2. **Increase memory limits** if encoding 4K content:
|
||||
```bash
|
||||
# Edit ~/.wslconfig
|
||||
[wsl2]
|
||||
memory=16GB
|
||||
```
|
||||
|
||||
## Customization
|
||||
|
||||
### Changing VMAF Targets
|
||||
|
||||
Edit `optimize_library.py`:
|
||||
|
||||
```python
|
||||
# More aggressive (smaller files, lower quality)
|
||||
TARGETS = [92.0, 90.0, 88.0]
|
||||
|
||||
# Conservative (larger files, higher quality)
|
||||
TARGETS = [95.0, 94.0, 93.0]
|
||||
```
|
||||
|
||||
### Changing Savings Threshold
|
||||
|
||||
```python
|
||||
# More aggressive (encode more)
|
||||
MIN_SAVINGS_PERCENT = 8.0
|
||||
|
||||
# Less aggressive (encode fewer)
|
||||
MIN_SAVINGS_PERCENT = 15.0
|
||||
```
|
||||
|
||||
### Changing Encoder Preset
|
||||
|
||||
```python
|
||||
# Faster encodes (larger files, lower quality)
|
||||
PRESET = 8
|
||||
|
||||
# Better quality (slower encodes, smaller files)
|
||||
PRESET = 4
|
||||
```
|
||||
|
||||
### Changing Estimate Target
|
||||
|
||||
```python
|
||||
# Target higher savings for estimates
|
||||
TARGET_SAVINGS_FOR_ESTIMATE = 20.0
|
||||
```
|
||||
|
||||
## File Structure
|
||||
|
||||
```
|
||||
/opt/Optmiser/
|
||||
├── optimize_library.py # Main encoding engine
|
||||
├── run_optimisation.sh # Linux/Server wrapper
|
||||
├── run_optimisation.ps1 # Windows wrapper
|
||||
├── bin/
|
||||
│ └── ab-av1 # ab-av1 binary
|
||||
├── tmp/ # Temporary encoding files
|
||||
├── logs/ # Log files (JSONL format)
|
||||
│ ├── tv_movies.jsonl
|
||||
│ ├── content.jsonl
|
||||
│ ├── low_savings_skips.jsonl
|
||||
│ ├── failed_searches.jsonl
|
||||
│ └── failed_encodes.jsonl
|
||||
├── .lock/ # Multi-machine coordination (created at runtime)
|
||||
├── README.md # This file
|
||||
├── SETUP.md # Setup instructions
|
||||
└── AGENTS.md # Technical documentation
|
||||
```
|
||||
|
||||
## Platform Support Matrix
|
||||
|
||||
| Platform | Status | Notes |
|
||||
|-----------|--------|-------|
|
||||
| Linux (Intel CPU) | ✅ Supported | Software encoding, multi-worker capable |
|
||||
| Windows (AMD GPU) | ✅ Supported | Hardware acceleration via d3d11va (auto-detects) |
|
||||
| Windows (Intel CPU) | ✅ Supported | Software encoding |
|
||||
| macOS (Apple Silicon) | ✅ Supported | Hardware via videotoolbox (auto-detects) |
|
||||
| WSL (Ubuntu/Debian) | ✅ Supported | Linux compatibility layer |
|
||||
| WSL (Windows drives) | ✅ Supported | Access via /mnt/c/ |
|
||||
|
||||
## Git Workflow
|
||||
|
||||
### Initial Setup
|
||||
|
||||
```bash
|
||||
cd /opt/Optmiser
|
||||
git init
|
||||
git remote add origin https://gitea.theflagroup.com/bnair/VMAFOptimiser.git
|
||||
git branch -M main
|
||||
git add .
|
||||
git commit -m "Initial commit: VMAF optimisation pipeline"
|
||||
git push -u origin main
|
||||
```
|
||||
|
||||
### Daily Updates
|
||||
|
||||
```bash
|
||||
cd /opt/Optmiser
|
||||
git pull origin main
|
||||
|
||||
# Run optimisation
|
||||
./run_optimisation.sh /media tv_movies
|
||||
|
||||
# Review changes
|
||||
git diff
|
||||
```
|
||||
|
||||
### Committing Changes
|
||||
|
||||
```bash
|
||||
cd /opt/Optmiser
|
||||
git status
|
||||
|
||||
# Add changed files
|
||||
git add optimize_library.py run_optimisation.sh run_optimisation.ps1
|
||||
|
||||
# Commit with message
|
||||
git commit -m "feat: add Windows and Linux wrapper scripts"
|
||||
|
||||
# Push
|
||||
git push
|
||||
```
|
||||
|
||||
### View History
|
||||
|
||||
```bash
|
||||
cd /opt/Optmiser
|
||||
git log --oneline
|
||||
git log --graph --all
|
||||
```
|
||||
|
||||
## FAQ
|
||||
|
||||
**Q: Can I run this on multiple machines at once?**
|
||||
A: Yes! Each machine will process different files due to lock file mechanism.
|
||||
|
||||
**Q: Should I use Windows or WSL?**
|
||||
A: WSL is recommended for Linux compatibility. Use Windows native if you need direct hardware access or performance.
|
||||
|
||||
**Q: Will hardware encoding work better than CPU?**
|
||||
A: For AMD RX 7900 XT, hardware AV1 encoding is ~3-10x faster than CPU. However, GPU encoding may need slightly lower VMAF targets to match quality.
|
||||
|
||||
**Q: What VMAF target should I use?**
|
||||
A: Start with VMAF 94 or 95. Drop to 92-90 if you need more savings.
|
||||
|
||||
**Q: How do I know which files are being processed?**
|
||||
A: Check `.lock/` directory: `ls -la /opt/Optmiser/.lock/`
|
||||
|
||||
**Q: Can I pause/resume?**
|
||||
A: Pause by stopping the script (Ctrl+C). Resume by running again - it skips processed files.
|
||||
|
||||
**Q: What happens if encoding fails?**
|
||||
A: Error is logged to `failed_encodes.jsonl`. Original file is NOT modified.
|
||||
|
||||
**Q: How much CPU does encoding use?**
|
||||
A: Full CPU by default. Use `--workers 1` for single-threaded, or limit with `--cpu-limit 50` for 50% (12 threads on 24-core).
|
||||
**Intelligent Video Library Optimization Pipeline**
|
||||
|
||||
Automatically optimizes your video library (Movies/TV) by finding the best compression (AV1/HEVC) that maintains a high visual quality target (VMAF 93+).
|
||||
|
||||
## Features
|
||||
|
||||
- **Hybrid Encoding:**
|
||||
- Uses **Hardware Acceleration** (NVIDIA NVENC, AMD AMF, Intel QSV, Apple VideoToolbox) for fast encoding.
|
||||
- Uses **Software Encoding** (CPU) as a robust fallback.
|
||||
- **VMAF Targeted:** Ensures visual quality matches the original (Target VMAF 93+).
|
||||
- **Multi-Platform:** Runs on **Windows**, **Linux**, and **macOS**.
|
||||
- **Distributed Processing:** Run multiple workers across multiple PCs sharing the same network library.
|
||||
- **Safety First:**
|
||||
- Locks files to prevent double-processing.
|
||||
- Verifies output size and integrity before replacing.
|
||||
- "Smart Resume" - skips already processed files.
|
||||
|
||||
---
|
||||
|
||||
**Last Updated:** December 31, 2025
|
||||
**Version:** 2.0 with Windows and Linux Wrapper Scripts
|
||||
## Directory Structure
|
||||
|
||||
```
|
||||
VMAFOptimiser/
|
||||
├── bin/ # Place ab-av1.exe here (or install to PATH)
|
||||
├── logs/ # Local logs (processed, rejected, stats)
|
||||
├── locks/ # Local lock files (if network not available)
|
||||
├── src/
|
||||
│ ├── smart_gpu_encoder.py # Main encoder engine
|
||||
│ ├── smart_monitor.py # TUI Dashboard (Watchdog)
|
||||
│ ├── vmaf_common.py # Shared logic
|
||||
│ └── smart_encoder.py # Legacy CPU-only engine
|
||||
├── run.ps1 # Windows One-Click Start
|
||||
└── README.md
|
||||
```
|
||||
|
||||
## Setup & Installation
|
||||
|
||||
### 1. Requirements
|
||||
|
||||
- **Python 3.10+**
|
||||
- **FFmpeg** (with libsvtav1/libx265 and HW drivers)
|
||||
- Windows: `choco install ffmpeg`
|
||||
- macOS: `brew install ffmpeg`
|
||||
- Linux: `sudo apt install ffmpeg`
|
||||
- **ab-av1** (VMAF calculator)
|
||||
- Download from [GitHub](https://github.com/alexheretic/ab-av1) and place in `bin/` OR install via `cargo install ab-av1`
|
||||
|
||||
### 2. Install Python Deps
|
||||
|
||||
```bash
|
||||
pip install watchdog rich
|
||||
```
|
||||
|
||||
### 3. Usage
|
||||
|
||||
#### Interactive Dashboard (Recommended)
|
||||
|
||||
Monitors directories and shows a TUI with progress bars.
|
||||
|
||||
```bash
|
||||
python src/smart_monitor.py --tv-dir "Z:\tv" --content-dir "Z:\content" --jobs 2
|
||||
```
|
||||
|
||||
#### Headless / Background (Cron/Task Scheduler)
|
||||
|
||||
Runs once through the library and exits.
|
||||
|
||||
```bash
|
||||
python src/smart_gpu_encoder.py --tv-dir "Z:\tv" --content-dir "Z:\content" --jobs 4
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Multi-PC Setup
|
||||
|
||||
To speed up processing, you can run this script on multiple computers simultaneously pointing to the same NAS/Network Share.
|
||||
|
||||
1. **Map Network Drive:** Ensure `Z:\` (or whatever path) is mapped on ALL computers.
|
||||
2. **Shared Locks:** The script automatically attempts to create a `.vmaf_locks` folder in the parent directory of your TV folder (e.g., `Z:\.vmaf_locks`).
|
||||
3. **Run:** Start the script on PC 1, PC 2, etc. They will respect each other's locks and not process the same file.
|
||||
|
||||
**Note:** Logs are stored LOCALLY on each PC in the `logs/` folder to prevent network file contention.
|
||||
|
||||
---
|
||||
|
||||
## Advanced Options
|
||||
|
||||
| Flag | Description |
|
||||
|------|-------------|
|
||||
| `--jobs N` | Number of parallel encoders (Default: 2) |
|
||||
| `--tv-only` | Scan only TV Shows |
|
||||
| `--content-only` | Scan only Movies |
|
||||
| `--skip-until "Name"` | Skip files until this keyword is found (good for resuming) |
|
||||
| `--debug` | Enable verbose logging |
|
||||
|
||||
---
|
||||
|
||||
## Credits
|
||||
|
||||
- Powered by `ab-av1` for VMAF calculation.
|
||||
- Uses `ffmpeg` for heavy lifting.
|
||||
|
||||
102
SETUP.md
102
SETUP.md
@@ -94,9 +94,12 @@ All wrapper scripts (`run_optimisation.sh` on Linux, `run_optimisation.ps1` on W
|
||||
| `--preset <value>` | SVT-AV1 Preset (4=best, 6=balanced, 8=fast) | 6 |
|
||||
| `--workers <count>` | Concurrent files to process | 1 |
|
||||
| `--samples <count>` | Samples for CRF search | 4 |
|
||||
| `--thorough` | Use thorough mode (slower, more accurate) | false |
|
||||
| `--encoder <name>` | ab-av1 encoder | svt-av1 |
|
||||
| `--hwaccel <value>` | Hardware acceleration | none (auto: auto-detect) |
|
||||
| `--encoder <name>` | Video encoder: svt-av1, av1_amf, av1_nvenc, av1_qsv | svt-av1 |
|
||||
| `--hwaccel <value>` | Hardware decode acceleration (auto, d3d11va, vaapi) | none |
|
||||
| `--use-hardware-worker` | Use 1 HW encoder worker + rest CPU workers | false |
|
||||
| `--plex-url <url>` | Plex server URL for library refresh | none |
|
||||
| `--plex-token <token>` | Plex auth token | none |
|
||||
| `--log-dir <path>` | Log directory | /opt/Optmiser/logs |
|
||||
|
||||
## Multi-Machine Setup
|
||||
|
||||
@@ -140,6 +143,99 @@ All three can run simultaneously!
|
||||
|
||||
## Hardware Acceleration
|
||||
|
||||
### Hardware Decoding vs Hardware Encoding
|
||||
|
||||
There are two types of hardware acceleration:
|
||||
|
||||
1. **Hardware Decoding (`--hwaccel`)**: Uses GPU to decode source video faster. Doesn't affect quality.
|
||||
2. **Hardware Encoding (`--encoder`)**: Uses GPU to encode output video. Much faster but requires quality compensation.
|
||||
|
||||
### Hardware Encoders
|
||||
|
||||
| Encoder | GPU | Platform | Notes |
|
||||
|---------|-----|----------|-------|
|
||||
| `svt-av1` | CPU | All | Default. Best quality, slowest. |
|
||||
| `av1_amf` | AMD | Windows/Linux | RX 7000 series, ~3-10x faster than CPU |
|
||||
| `av1_nvenc` | NVIDIA | Windows/Linux | RTX 40 series, very fast |
|
||||
| `av1_qsv` | Intel | Windows/Linux | Arc GPUs, Intel iGPU with AV1 |
|
||||
|
||||
### FFmpeg with Hardware Encoder Support
|
||||
|
||||
**Windows (pre-built with HW encoders):**
|
||||
```powershell
|
||||
# Most Windows FFmpeg builds include hardware encoders
|
||||
# Verify with:
|
||||
ffmpeg -encoders 2>&1 | findstr av1
|
||||
|
||||
# Should show: av1_amf, av1_nvenc, av1_qsv (depending on your GPU)
|
||||
|
||||
# If missing, download from: https://github.com/BtbN/FFmpeg-Builds/releases
|
||||
# Choose: ffmpeg-master-latest-win64-gpl.zip (includes all encoders)
|
||||
```
|
||||
|
||||
**Linux (may need custom build):**
|
||||
```bash
|
||||
# Check available encoders
|
||||
ffmpeg -encoders 2>&1 | grep av1
|
||||
|
||||
# For AMD (av1_amf) - requires AMF SDK
|
||||
# Install AMD GPU drivers with AMF support
|
||||
sudo apt install amdgpu-pro
|
||||
|
||||
# For NVIDIA (av1_nvenc) - requires NVIDIA drivers
|
||||
sudo apt install nvidia-driver-535 # or newer
|
||||
|
||||
# For Intel (av1_qsv) - requires Intel Media SDK
|
||||
sudo apt install intel-media-va-driver-non-free
|
||||
|
||||
# If your distro's ffmpeg lacks HW encoders, use static build:
|
||||
wget https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz
|
||||
```
|
||||
|
||||
### Using Hardware Workers (1 GPU + 3 CPU)
|
||||
|
||||
For mixed encoding with your RX 7900 XT:
|
||||
|
||||
```powershell
|
||||
# Windows - 4 workers: 1 uses GPU (av1_amf), 3 use CPU (svt-av1)
|
||||
.\run_optimisation.ps1 `
|
||||
-Directory "Z:\" `
|
||||
-Vmaf 94 `
|
||||
-Workers 4 `
|
||||
-Encoder av1_amf `
|
||||
-Hwaccel auto `
|
||||
-UseHardwareWorker `
|
||||
-LogDir "C:\Users\bnair\Documents\VMAFOptimiser\logs"
|
||||
```
|
||||
|
||||
```bash
|
||||
# Linux - same concept
|
||||
./run_optimisation.sh \
|
||||
--directory /media \
|
||||
--vmaf 94 \
|
||||
--workers 4 \
|
||||
--encoder av1_amf \
|
||||
--hwaccel auto \
|
||||
--use-hardware-worker
|
||||
```
|
||||
|
||||
**How it works:**
|
||||
- First worker to start claims the GPU and uses `av1_amf`
|
||||
- Remaining 3 workers use `svt-av1` (CPU)
|
||||
- GPU worker applies +2 VMAF offset automatically to match CPU quality
|
||||
|
||||
### Quality Compensation
|
||||
|
||||
Hardware encoders produce lower quality at the same settings. The script automatically compensates:
|
||||
|
||||
| Target VMAF | CPU (svt-av1) | GPU (av1_amf) |
|
||||
|-------------|---------------|---------------|
|
||||
| 94 | Searches for VMAF 94 | Searches for VMAF 96 |
|
||||
| 93 | Searches for VMAF 93 | Searches for VMAF 95 |
|
||||
| 92 | Searches for VMAF 92 | Searches for VMAF 94 |
|
||||
|
||||
This offset (`HW_ENCODER_VMAF_OFFSET = 2.0`) can be adjusted in `optimize_library.py`.
|
||||
|
||||
### Automatic Detection
|
||||
|
||||
When `--hwaccel auto` is specified, the wrapper scripts automatically select the best available hardware acceleration:
|
||||
|
||||
BIN
bin/ab-av1.exe
Normal file
BIN
bin/ab-av1.exe
Normal file
Binary file not shown.
51
debug_status.py
Normal file
51
debug_status.py
Normal file
@@ -0,0 +1,51 @@
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
# Paths
|
||||
Z_LOGS = Path(r"Z:\.vmaf_logs")
|
||||
C_LOGS = Path(r"C:\Users\bnair\Videos\encodes\logs")
|
||||
LOCAL_LOGS = Path("logs").resolve()
|
||||
Z_LOCKS = Path(r"Z:\.vmaf_locks")
|
||||
C_LOCKS = Path(r"C:\Users\bnair\Videos\encodes\locks")
|
||||
|
||||
print("--- DEBUG STATUS ---")
|
||||
|
||||
print(f"\n1. Checking Logs on Z: ({Z_LOGS})")
|
||||
if Z_LOGS.exists():
|
||||
print(" [OK] Folder exists.")
|
||||
for f in Z_LOGS.glob("*"):
|
||||
print(f" - {f.name} ({f.stat().st_size} bytes)")
|
||||
else:
|
||||
print(" [MISSING] Folder does not exist.")
|
||||
|
||||
print(f"\n2. Checking Logs on C: ({C_LOGS})")
|
||||
if C_LOGS.exists():
|
||||
print(" [OK] Folder exists.")
|
||||
for f in C_LOGS.glob("*"):
|
||||
print(f" - {f.name} ({f.stat().st_size} bytes)")
|
||||
else:
|
||||
print(" [MISSING] Folder does not exist.")
|
||||
|
||||
print(f"\n3. Checking Local Logs: ({LOCAL_LOGS})")
|
||||
if LOCAL_LOGS.exists():
|
||||
print(" [OK] Folder exists.")
|
||||
for f in LOCAL_LOGS.glob("*"):
|
||||
print(f" - {f.name} ({f.stat().st_size} bytes)")
|
||||
else:
|
||||
print(" [MISSING] Folder does not exist.")
|
||||
|
||||
print(f"\n4. Checking Locks")
|
||||
z_count = len(list(Z_LOCKS.glob("*"))) if Z_LOCKS.exists() else 0
|
||||
c_count = len(list(C_LOCKS.glob("*"))) if C_LOCKS.exists() else 0
|
||||
print(f" Z: Locks: {z_count}")
|
||||
print(f" C: Locks: {c_count}")
|
||||
|
||||
print(f"\n4. Checking Temp Encodes")
|
||||
temp = Path(r"C:\Users\bnair\Videos\encodes")
|
||||
if temp.exists():
|
||||
mkv_files = list(temp.glob("*.mkv"))
|
||||
print(f" Found {len(mkv_files)} mkv files.")
|
||||
for f in mkv_files:
|
||||
print(f" - {f.name} ({f.stat().st_size / 1024**3:.2f} GB)")
|
||||
else:
|
||||
print(" [MISSING] Temp folder missing.")
|
||||
207
legacy/finalOptimiser.ps1
Normal file
207
legacy/finalOptimiser.ps1
Normal file
@@ -0,0 +1,207 @@
|
||||
# SMART PRE-FLIGHT ENCODER - VERBOSE CONSOLE OUTPUT
|
||||
# Usage: .\Encode-TVShows.ps1
|
||||
|
||||
param(
|
||||
[string]$TvDir = "Z:\tv",
|
||||
[string]$ContentDir = "Z:\content",
|
||||
[int]$MaxJobs = 2,
|
||||
[int]$Av1Q = 34,
|
||||
[int]$HevcQ = 28,
|
||||
[string]$EncoderAV1 = "av1_amf",
|
||||
[string]$EncoderHEVC = "hevc_amf",
|
||||
[switch]$SkipAV1 = $true
|
||||
)
|
||||
|
||||
# --- CONFIGURATION ---
|
||||
$Global:FFMPEG = "ffmpeg"
|
||||
$Global:FFPROBE = "ffprobe"
|
||||
$Global:TEMP_DIR = "C:\Users\bnair\Videos\encodes"
|
||||
$Global:LockDir = "C:\Users\bnair\Videos\encodes\locks"
|
||||
$LogFileTV = "C:\Users\bnair\Videos\encodes\encoding-log-tv.csv"
|
||||
$LogFileContent = "C:\Users\bnair\Videos\encodes\encoding-log-content.csv"
|
||||
|
||||
if (-not (Test-Path $Global:TEMP_DIR)) { New-Item -ItemType Directory -Path $Global:TEMP_DIR -Force | Out-Null }
|
||||
if (-not (Test-Path $Global:LockDir)) { New-Item -ItemType Directory -Path $Global:LockDir -Force | Out-Null }
|
||||
|
||||
function Init-LogFile {
|
||||
param([string]$Path)
|
||||
if (-not (Test-Path $Path)) { "Timestamp,File,InputSize,OutputSize,CodecUsed,Status,Savings" | Out-File -FilePath $Path -Encoding UTF8 }
|
||||
}
|
||||
Init-LogFile $LogFileTV
|
||||
Init-LogFile $LogFileContent
|
||||
|
||||
function Test-Tools {
|
||||
Write-Host "Setup: Checking required tools..." -ForegroundColor Cyan
|
||||
if (-not (Get-Command $Global:FFMPEG -ErrorAction SilentlyContinue)) { Write-Host "ERROR: ffmpeg not found!" -ForegroundColor Red; exit 1 }
|
||||
Write-Host "Setup: Tools found." -ForegroundColor Green
|
||||
}
|
||||
|
||||
$SharedFunctions = {
|
||||
function Get-LockId {
|
||||
param([string]$FilePath)
|
||||
try {
|
||||
$pathBytes = [System.Text.Encoding]::UTF8.GetBytes($FilePath)
|
||||
$hash = [System.Security.Cryptography.SHA256]::Create().ComputeHash($pathBytes)
|
||||
return [BitConverter]::ToString($hash).Replace("-", "").Substring(0, 16)
|
||||
} catch { return "unknown_lock" }
|
||||
}
|
||||
|
||||
function Run-FFmpeg {
|
||||
param($In, $Out, $Enc, $Q, $Seek=$null, $Duration=$null)
|
||||
$argsList = "-hide_banner -loglevel error -y"
|
||||
if ($Seek) { $argsList += " -ss $Seek" }
|
||||
$argsList += " -i `"$In`""
|
||||
if ($Duration) { $argsList += " -t $Duration" }
|
||||
$argsList += " -c:v $Enc -usage transcoding -quality quality -rc cqp -qp_i $Q -qp_p $Q -qp_b $Q -c:a copy `"$Out`""
|
||||
return Start-Process -FilePath "ffmpeg" -ArgumentList $argsList -NoNewWindow -Wait -PassThru
|
||||
}
|
||||
|
||||
function Process-VideoFile {
|
||||
param($InputFile, $CurrentLogFile, $LockDir, $TempDir, $Av1Q, $HevcQ, $EncAV1, $EncHEVC, $SkipAV1)
|
||||
|
||||
$pidStr = $PID.ToString()
|
||||
$lockId = Get-LockId -FilePath $InputFile
|
||||
$lockFile = Join-Path $LockDir "$lockId.lock"
|
||||
|
||||
try {
|
||||
if (Test-Path $lockFile) { return }
|
||||
$pidStr | Out-File -FilePath $lockFile -Force
|
||||
|
||||
$fileName = Split-Path $InputFile -Leaf
|
||||
Write-Host "[$pidStr] Found: $fileName" -ForegroundColor White
|
||||
|
||||
# Skip Logic
|
||||
$currentCodec = (& "ffprobe" -v error -select_streams v:0 -show_entries stream=codec_name -of csv=p=0 "$InputFile" 2>&1)
|
||||
if ($SkipAV1 -and ($currentCodec -match "av1" -or $currentCodec -match "hevc")) {
|
||||
Write-Host "[$pidStr] SKIP: Already optimized ($currentCodec)" -ForegroundColor DarkGray
|
||||
return
|
||||
}
|
||||
|
||||
$inputSize = (Get-Item $InputFile).Length
|
||||
|
||||
# --- PHASE 1: PRE-FLIGHT SAMPLE ---
|
||||
Write-Host "[$pidStr] Testing: Generating 60s sample..." -ForegroundColor Yellow
|
||||
$tempSample = Join-Path $TempDir "$fileName.sample.mkv"
|
||||
$procSample = Run-FFmpeg $InputFile $tempSample $EncAV1 $Av1Q "00:05:00" "60"
|
||||
|
||||
$doFullAV1 = $true
|
||||
|
||||
if ($procSample.ExitCode -eq 0 -and (Test-Path $tempSample)) {
|
||||
$sampleSize = (Get-Item $tempSample).Length
|
||||
# Threshold: 150MB for 60s is ~20Mbps (Likely bloat)
|
||||
if ($sampleSize -gt 150MB) {
|
||||
Write-Host "[$pidStr] Test Result: FAIL. Sample was $([math]::Round($sampleSize/1MB))MB. Too big for AV1." -ForegroundColor Red
|
||||
$doFullAV1 = $false
|
||||
} else {
|
||||
Write-Host "[$pidStr] Test Result: PASS. Sample was $([math]::Round($sampleSize/1MB))MB. Proceeding with AV1." -ForegroundColor Green
|
||||
}
|
||||
Remove-Item $tempSample -Force
|
||||
}
|
||||
|
||||
$finalStatus = "Failed"
|
||||
$finalCodec = "None"
|
||||
$finalSize = 0
|
||||
$finalSavings = 0.00
|
||||
|
||||
# --- PHASE 2: FULL AV1 ---
|
||||
if ($doFullAV1) {
|
||||
Write-Host "[$pidStr] Action: Starting Full AV1 Encode..." -ForegroundColor Cyan
|
||||
$tempAV1 = Join-Path $TempDir "$fileName.av1.mkv"
|
||||
$procAV1 = Run-FFmpeg $InputFile $tempAV1 $EncAV1 $Av1Q
|
||||
|
||||
if ($procAV1.ExitCode -eq 0 -and (Test-Path $tempAV1)) {
|
||||
$sizeAV1 = (Get-Item $tempAV1).Length
|
||||
|
||||
if ($sizeAV1 -lt $inputSize) {
|
||||
$finalSavings = [math]::Round((1 - ($sizeAV1 / $inputSize)) * 100, 2)
|
||||
Write-Host "[$pidStr] AV1 ACCEPTED: Saved ${finalSavings}%" -ForegroundColor Green
|
||||
|
||||
$finalOut = $InputFile -replace '\.mkv$', '_av1.mkv' -replace '\.mp4$', '_av1.mp4'
|
||||
Move-Item $tempAV1 -Destination $finalOut -Force
|
||||
Remove-Item $InputFile -Force
|
||||
Rename-Item $finalOut -NewName $fileName -Force
|
||||
|
||||
"$(Get-Date -Format 'yyyy-MM-dd HH:mm:ss'),`"$InputFile`",$inputSize,$sizeAV1,AV1,`"Replaced (AV1)`",$finalSavings%" | Out-File -FilePath $CurrentLogFile -Append -Encoding UTF8
|
||||
return
|
||||
} else {
|
||||
Write-Host "[$pidStr] AV1 REJECTED: Larger than source ($([math]::Round($sizeAV1/1GB,2)) GB). Deleting..." -ForegroundColor Red
|
||||
Remove-Item $tempAV1 -Force
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# --- PHASE 3: HEVC FALLBACK ---
|
||||
Write-Host "[$pidStr] Action: Trying HEVC Fallback..." -ForegroundColor Cyan
|
||||
$tempHEVC = Join-Path $TempDir "$fileName.hevc.mkv"
|
||||
$procHEVC = Run-FFmpeg $InputFile $tempHEVC $EncHEVC $HevcQ
|
||||
|
||||
if ($procHEVC.ExitCode -eq 0 -and (Test-Path $tempHEVC)) {
|
||||
$sizeHEVC = (Get-Item $tempHEVC).Length
|
||||
|
||||
if ($sizeHEVC -lt $inputSize) {
|
||||
$finalSavings = [math]::Round((1 - ($sizeHEVC / $inputSize)) * 100, 2)
|
||||
Write-Host "[$pidStr] HEVC ACCEPTED: Saved ${finalSavings}%" -ForegroundColor Green
|
||||
|
||||
$finalOut = $InputFile -replace '\.mkv$', '_hevc.mkv' -replace '\.mp4$', '_hevc.mp4'
|
||||
Move-Item $tempHEVC -Destination $finalOut -Force
|
||||
Remove-Item $InputFile -Force
|
||||
Rename-Item $finalOut -NewName $fileName -Force
|
||||
|
||||
$finalStatus = "Replaced (HEVC)"
|
||||
$finalCodec = "HEVC"
|
||||
$finalSize = $sizeHEVC
|
||||
} else {
|
||||
Write-Host "[$pidStr] HEVC REJECTED: Also larger. Keeping original." -ForegroundColor Red
|
||||
Remove-Item $tempHEVC -Force
|
||||
$finalStatus = "Rejected (Both Larger)"
|
||||
$finalSize = $sizeHEVC
|
||||
$finalCodec = "HEVC"
|
||||
}
|
||||
}
|
||||
|
||||
"$(Get-Date -Format 'yyyy-MM-dd HH:mm:ss'),`"$InputFile`",$inputSize,$finalSize,$finalCodec,`"$finalStatus`",$finalSavings%" | Out-File -FilePath $CurrentLogFile -Append -Encoding UTF8
|
||||
|
||||
} finally {
|
||||
Remove-Item $lockFile -Force -ErrorAction SilentlyContinue
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function Process-Directory {
|
||||
param($TargetDirectory, $TargetLogFile, $PhaseName)
|
||||
if (-not (Test-Path $TargetDirectory)) { return }
|
||||
|
||||
Write-Host "`n=== PHASE: $PhaseName ===" -ForegroundColor Magenta
|
||||
$files = Get-ChildItem -Path $TargetDirectory -Include *.mkv, *.mp4 -Recurse -File -ErrorAction SilentlyContinue
|
||||
$videoFiles = $files | Where-Object { $_.Name -notmatch "_av1" -and $_.Name -notmatch "_hevc" }
|
||||
|
||||
Write-Host "Found $($videoFiles.Count) files." -ForegroundColor Cyan
|
||||
|
||||
$processed = 0
|
||||
while ($processed -lt $videoFiles.Count) {
|
||||
$batchSize = [math]::Min($MaxJobs, ($videoFiles.Count - $processed))
|
||||
$currentBatch = $videoFiles[$processed..($processed + $batchSize - 1)]
|
||||
$jobs = @()
|
||||
|
||||
foreach ($file in $currentBatch) {
|
||||
$jobs += Start-Job -InitializationScript $SharedFunctions -ScriptBlock {
|
||||
param($f, $log, $lock, $temp, $av1q, $hevcq, $e1, $e2, $skip)
|
||||
Process-VideoFile $f $log $lock $temp $av1q $hevcq $e1 $e2 $skip
|
||||
} -ArgumentList $file.FullName, $TargetLogFile, $Global:LockDir, $Global:TEMP_DIR, $Av1Q, $HevcQ, $EncoderAV1, $EncoderHEVC, $SkipAV1
|
||||
}
|
||||
|
||||
while (($jobs | Where-Object { $_.State -eq 'Running' }).Count -gt 0) {
|
||||
$jobs | Receive-Job
|
||||
Start-Sleep -Seconds 2
|
||||
}
|
||||
$jobs | Receive-Job
|
||||
$jobs | Remove-Job -Force
|
||||
$processed += $batchSize
|
||||
Write-Host "Progress Phase ${PhaseName}: $processed / $($videoFiles.Count)" -ForegroundColor Yellow
|
||||
}
|
||||
}
|
||||
|
||||
Test-Tools
|
||||
Process-Directory $TvDir $LogFileTV "TV-Shows"
|
||||
Process-Directory $ContentDir $LogFileContent "Content"
|
||||
Write-Host "`nDone." -ForegroundColor Green
|
||||
931
legacy/optimize_library.py
Normal file
931
legacy/optimize_library.py
Normal file
@@ -0,0 +1,931 @@
|
||||
import os
|
||||
import sys
|
||||
import subprocess
|
||||
import argparse
|
||||
import json
|
||||
import shutil
|
||||
import platform
|
||||
import time
|
||||
import signal
|
||||
from pathlib import Path
|
||||
from datetime import datetime
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed, ProcessPoolExecutor
|
||||
from threading import Lock, get_ident
|
||||
|
||||
# --- Configuration ---
|
||||
DEFAULT_VMAF = 95.0
|
||||
DEFAULT_PRESET = 6
|
||||
DEFAULT_WORKERS = 1
|
||||
DEFAULT_SAMPLES = 4
|
||||
EXTENSIONS = {".mkv", ".mp4", ".mov", ".avi", ".ts"}
|
||||
|
||||
TARGETS = [94.0, 93.0, 92.0, 90.0]
|
||||
MIN_SAVINGS_PERCENT = 12.0
|
||||
TARGET_SAVINGS_FOR_ESTIMATE = 15.0
|
||||
|
||||
# Hardware encoder quality compensation (GPU needs higher VMAF target to match CPU quality)
|
||||
HW_ENCODER_VMAF_OFFSET = 2.0
|
||||
HW_ENCODERS = {"av1_amf", "av1_nvenc", "av1_qsv", "av1_vaapi"}
|
||||
|
||||
# Global state for resume capability
|
||||
_processed_files = set()
|
||||
_lock = Lock()
|
||||
_shutdown_requested = False
|
||||
_AB_AV1_HELP_CACHE = {}
|
||||
_hw_worker_id = None # Thread ID of the designated hardware worker
|
||||
|
||||
|
||||
def claim_hardware_worker():
|
||||
"""Attempt to claim hardware worker status for this thread. Returns True if claimed."""
|
||||
global _hw_worker_id
|
||||
with _lock:
|
||||
thread_id = get_ident()
|
||||
if _hw_worker_id is None:
|
||||
_hw_worker_id = thread_id
|
||||
return True
|
||||
return _hw_worker_id == thread_id
|
||||
|
||||
|
||||
def signal_handler(signum, frame):
|
||||
"""Handle graceful shutdown"""
|
||||
global _shutdown_requested
|
||||
_shutdown_requested = True
|
||||
print("\n\n⚠️ Shutdown requested. Finishing current tasks...")
|
||||
|
||||
|
||||
signal.signal(signal.SIGINT, signal_handler)
|
||||
signal.signal(signal.SIGTERM, signal_handler)
|
||||
|
||||
|
||||
def check_dependencies():
|
||||
missing = []
|
||||
for tool in ["ffmpeg", "ffprobe", "ab-av1"]:
|
||||
if not shutil.which(tool):
|
||||
missing.append(tool)
|
||||
|
||||
if missing:
|
||||
print(f"Error: Missing required tools: {', '.join(missing)}")
|
||||
print(
|
||||
"Please install FFmpeg and 'ab-av1' (via cargo install ab-av1) before running."
|
||||
)
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
def is_wsl():
|
||||
if os.environ.get("WSL_DISTRO_NAME"):
|
||||
return True
|
||||
|
||||
try:
|
||||
with open("/proc/sys/kernel/osrelease", "r", encoding="utf-8") as f:
|
||||
return "microsoft" in f.read().lower()
|
||||
except FileNotFoundError:
|
||||
return False
|
||||
|
||||
|
||||
def platform_label():
|
||||
system = platform.system()
|
||||
if system == "Linux" and is_wsl():
|
||||
return "Linux (WSL)"
|
||||
return system
|
||||
|
||||
|
||||
def _ab_av1_help(subcommand):
|
||||
cached = _AB_AV1_HELP_CACHE.get(subcommand)
|
||||
if cached is not None:
|
||||
return cached
|
||||
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["ab-av1", subcommand, "--help"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=False,
|
||||
)
|
||||
help_text = (result.stdout or "") + "\n" + (result.stderr or "")
|
||||
except Exception:
|
||||
help_text = ""
|
||||
|
||||
_AB_AV1_HELP_CACHE[subcommand] = help_text
|
||||
return help_text
|
||||
|
||||
|
||||
def ab_av1_supports(subcommand, flag):
|
||||
return flag in _ab_av1_help(subcommand)
|
||||
|
||||
|
||||
def normalize_hwaccel(value):
|
||||
if value is None:
|
||||
return None
|
||||
|
||||
v = value.strip()
|
||||
if not v:
|
||||
return None
|
||||
|
||||
v_lower = v.lower()
|
||||
if v_lower in {"none", "off", "false", "0"}:
|
||||
return None
|
||||
|
||||
if v_lower != "auto":
|
||||
return v
|
||||
|
||||
system = platform.system()
|
||||
if system == "Windows":
|
||||
return "d3d11va"
|
||||
if system == "Darwin":
|
||||
return "videotoolbox"
|
||||
|
||||
return "vaapi"
|
||||
|
||||
|
||||
def get_probe_data(filepath):
|
||||
"""Get comprehensive video data using ffprobe"""
|
||||
try:
|
||||
cmd = [
|
||||
"ffprobe",
|
||||
"-v",
|
||||
"quiet",
|
||||
"-print_format",
|
||||
"json",
|
||||
"-show_streams",
|
||||
"-show_format",
|
||||
str(filepath),
|
||||
]
|
||||
# FIX: Added encoding="utf-8" and errors="ignore"
|
||||
res = subprocess.run(
|
||||
cmd,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=True,
|
||||
encoding="utf-8",
|
||||
errors="ignore"
|
||||
)
|
||||
return json.loads(res.stdout)
|
||||
except Exception as e:
|
||||
print(f"Error probing {filepath}: {e}")
|
||||
return None
|
||||
|
||||
|
||||
def get_video_stats(data):
|
||||
"""Extract video statistics from ffprobe output"""
|
||||
if not data or "streams" not in data or "format" not in data:
|
||||
return None
|
||||
|
||||
v_stream = next((s for s in data["streams"] if s["codec_type"] == "video"), None)
|
||||
if not v_stream:
|
||||
return None
|
||||
|
||||
size = int(data["format"].get("size", 0))
|
||||
duration = float(data["format"].get("duration", 0))
|
||||
|
||||
# Calculate bitrate from file size and duration (more reliable than ffprobe's bitrate)
|
||||
if size > 0 and duration > 0:
|
||||
bitrate = int((size * 8) / duration / 1000)
|
||||
else:
|
||||
bitrate = int(data["format"].get("bitrate", 0)) // 1000
|
||||
|
||||
return {
|
||||
"codec": v_stream.get("codec_name"),
|
||||
"width": v_stream.get("width"),
|
||||
"height": v_stream.get("height"),
|
||||
"bitrate": bitrate,
|
||||
"size": size,
|
||||
"duration": duration,
|
||||
}
|
||||
|
||||
|
||||
def log_result(log_dir, log_name, data):
|
||||
"""Log result to JSONL file"""
|
||||
os.makedirs(log_dir, exist_ok=True)
|
||||
log_file = Path(log_dir) / f"{log_name}.jsonl"
|
||||
data["timestamp"] = datetime.now().isoformat()
|
||||
|
||||
with _lock:
|
||||
with open(log_file, "a") as f:
|
||||
f.write(json.dumps(data) + "\n")
|
||||
|
||||
|
||||
def run_command_streaming(cmd, description=""):
|
||||
"""Run command and stream output in real-time"""
|
||||
print(f" [Running {description}]")
|
||||
process = subprocess.Popen(
|
||||
cmd,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.STDOUT,
|
||||
universal_newlines=True,
|
||||
bufsize=1,
|
||||
)
|
||||
|
||||
if process.stdout:
|
||||
for line in process.stdout:
|
||||
if _shutdown_requested:
|
||||
process.terminate()
|
||||
break
|
||||
print(f" {line.rstrip()}")
|
||||
|
||||
process.wait()
|
||||
return process.returncode
|
||||
|
||||
|
||||
def run_crf_search(
|
||||
filepath,
|
||||
target_vmaf,
|
||||
preset,
|
||||
temp_dir,
|
||||
encoder="svt-av1",
|
||||
use_hw=False,
|
||||
hwaccel=None,
|
||||
):
|
||||
"""Run CRF search for a specific VMAF target"""
|
||||
is_hw_encoder = encoder in HW_ENCODERS
|
||||
|
||||
# Apply VMAF offset for hardware encoders to match software quality
|
||||
effective_vmaf = (
|
||||
target_vmaf + HW_ENCODER_VMAF_OFFSET if is_hw_encoder else target_vmaf
|
||||
)
|
||||
|
||||
# --- Build the Command (Used for both running and parsing) ---
|
||||
cmd = [
|
||||
"ab-av1",
|
||||
"crf-search",
|
||||
"-i",
|
||||
str(filepath),
|
||||
"--min-vmaf",
|
||||
str(effective_vmaf),
|
||||
"--preset",
|
||||
str(preset),
|
||||
"--max-encoded-percent",
|
||||
"100",
|
||||
"--temp-dir",
|
||||
temp_dir,
|
||||
"--samples",
|
||||
"4",
|
||||
]
|
||||
|
||||
# FORCE use of the specified encoder (Fixes the CPU default crash)
|
||||
if ab_av1_supports("crf-search", "--encoder"):
|
||||
cmd.extend(["--encoder", encoder])
|
||||
|
||||
# Hardware decode acceleration
|
||||
if use_hw and hwaccel:
|
||||
if ab_av1_supports("crf-search", "--enc-input"):
|
||||
cmd.extend(["--enc-input", f"hwaccel={hwaccel}"])
|
||||
if hwaccel == "vaapi":
|
||||
cmd.extend(["--enc-input", "hwaccel_output_format=vaapi"])
|
||||
|
||||
vmaf_label = f"VMAF {effective_vmaf}" if is_hw_encoder else f"VMAF {target_vmaf}"
|
||||
print(f" - Searching for CRF to hit {vmaf_label}...")
|
||||
|
||||
# 1. First Run: Stream output to console
|
||||
returncode = run_command_streaming(cmd, f"crf-search {vmaf_label}")
|
||||
|
||||
if returncode == 0:
|
||||
# 2. Second Run: Capture output to parse the CRF
|
||||
# FIX: Reuse 'cmd' and add encoding parameters to prevent charmap crashes
|
||||
res = subprocess.run(
|
||||
cmd,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
encoding="utf-8",
|
||||
errors="ignore"
|
||||
)
|
||||
|
||||
lines = res.stdout.strip().split("\n")
|
||||
for line in reversed(lines):
|
||||
if "crf" in line.lower():
|
||||
try:
|
||||
parts = line.split()
|
||||
crf_val = float(parts[1])
|
||||
percent = 100.0
|
||||
for p in parts:
|
||||
if "%" in p:
|
||||
percent = float(p.strip("()%"))
|
||||
break
|
||||
return {
|
||||
"crf": crf_val,
|
||||
"predicted_percent": percent,
|
||||
"vmaf": target_vmaf,
|
||||
}
|
||||
except Exception as e:
|
||||
print(f" ! Failed to parse crf-search output: {e}")
|
||||
return None
|
||||
|
||||
|
||||
def find_target_savings_params(
|
||||
filepath,
|
||||
start_vmaf,
|
||||
preset,
|
||||
temp_dir,
|
||||
encoder="svt-av1",
|
||||
use_hw=False,
|
||||
hwaccel=None,
|
||||
):
|
||||
"""Find VMAF target that achieves minimum savings"""
|
||||
print(f"\n --- Finding VMAF for {TARGET_SAVINGS_FOR_ESTIMATE}% savings ---")
|
||||
|
||||
test_targets = [t for t in TARGETS if t <= start_vmaf]
|
||||
|
||||
for i, target in enumerate(test_targets):
|
||||
if _shutdown_requested:
|
||||
return None
|
||||
|
||||
print(
|
||||
f" Testing VMAF {target} for {TARGET_SAVINGS_FOR_ESTIMATE}% target... (test {i + 1}/{len(test_targets)})"
|
||||
)
|
||||
result = run_crf_search(
|
||||
filepath, target, preset, temp_dir, encoder, use_hw, hwaccel
|
||||
)
|
||||
|
||||
if result:
|
||||
predicted_savings = 100.0 - result["predicted_percent"]
|
||||
quality_drop = start_vmaf - target
|
||||
print(
|
||||
f" ✓ VMAF {target}: CRF {result['crf']}, Savings {predicted_savings:.1f}%, Drop -{quality_drop:.0f}"
|
||||
)
|
||||
|
||||
if predicted_savings >= TARGET_SAVINGS_FOR_ESTIMATE:
|
||||
print(f"\n ✅ FOUND {TARGET_SAVINGS_FOR_ESTIMATE}%+ SAVINGS:")
|
||||
print(f" Target VMAF: {target} (quality drop: -{quality_drop:.0f})")
|
||||
print(f" CRF: {result['crf']}")
|
||||
print(f" Predicted savings: {predicted_savings:.1f}%")
|
||||
return {
|
||||
"target_vmaf": target,
|
||||
"crf": result["crf"],
|
||||
"savings": predicted_savings,
|
||||
"quality_drop": quality_drop,
|
||||
"found": True,
|
||||
}
|
||||
else:
|
||||
print(f" ✗ Could not achieve VMAF {target}")
|
||||
|
||||
print(f"\n 📝 COULD NOT ACHIEVE {TARGET_SAVINGS_FOR_ESTIMATE}% SAVINGS")
|
||||
print(f" Tried VMAF targets: {test_targets}")
|
||||
return None
|
||||
|
||||
|
||||
def run_encode(
|
||||
filepath, output_path, crf, preset, encoder="svt-av1", use_hw=False, hwaccel=None
|
||||
):
|
||||
"""Run full encoding with real-time output streaming"""
|
||||
cmd = [
|
||||
"ab-av1",
|
||||
"encode",
|
||||
"--input",
|
||||
str(filepath),
|
||||
"--output",
|
||||
str(output_path),
|
||||
"--crf",
|
||||
str(crf),
|
||||
"--preset",
|
||||
str(preset),
|
||||
"--acodec",
|
||||
"copy",
|
||||
]
|
||||
|
||||
if encoder != "svt-av1":
|
||||
if ab_av1_supports("encode", "--encoder"):
|
||||
cmd.extend(["--encoder", encoder])
|
||||
|
||||
if use_hw and hwaccel:
|
||||
if ab_av1_supports("encode", "--enc-input"):
|
||||
cmd.extend(["--enc-input", f"hwaccel={hwaccel}"])
|
||||
if hwaccel == "vaapi":
|
||||
cmd.extend(["--enc-input", "hwaccel_output_format=vaapi"])
|
||||
|
||||
return run_command_streaming(cmd, f"encoding (CRF {crf})")
|
||||
|
||||
|
||||
def provide_recommendations(
|
||||
stats_before, hit_vmaf, predicted_savings, target_result=None
|
||||
):
|
||||
"""Provide recommendations based on analysis results"""
|
||||
print(f"\n --- Recommendations for {stats_before['codec']} content ---")
|
||||
|
||||
if target_result and target_result["found"]:
|
||||
print(f" 📊 TO HIT {TARGET_SAVINGS_FOR_ESTIMATE}% SAVINGS:")
|
||||
print(
|
||||
f" → Target VMAF: {target_result['target_vmaf']} (drop: -{target_result['quality_drop']:.0f})"
|
||||
)
|
||||
print(f" → CRF: {target_result['crf']}")
|
||||
print(f" → Predicted: {target_result['savings']:.1f}% savings")
|
||||
print(f" → Trade-off: Quality reduction for space savings")
|
||||
print()
|
||||
|
||||
if stats_before["bitrate"] < 2000:
|
||||
print(f" → Source bitrate is low ({stats_before['bitrate']}k)")
|
||||
print(f" → AV1 gains minimal on highly compressed sources")
|
||||
print(f" → Recommendation: SKIP - Source already optimized")
|
||||
return
|
||||
|
||||
if stats_before["codec"] in ["hevc", "h265", "vp9"]:
|
||||
print(f" → Source already uses modern codec ({stats_before['codec']})")
|
||||
print(f" → AV1 gains minimal on already-compressed content")
|
||||
print(f" → Recommendation: SKIP - Codec already efficient")
|
||||
return
|
||||
|
||||
if target_result and not target_result["found"]:
|
||||
print(
|
||||
f" → Could not achieve {TARGET_SAVINGS_FOR_ESTIMATE}% even at lowest VMAF"
|
||||
)
|
||||
print(f" → Content may not compress well with AV1")
|
||||
print(f" → Recommendation: SKIP - Review manually")
|
||||
|
||||
|
||||
def refresh_plex(plex_url, plex_token):
|
||||
"""Refresh Plex library after encoding completion"""
|
||||
if not plex_url or not plex_token:
|
||||
return
|
||||
|
||||
try:
|
||||
print("\n📡 Refreshing Plex library...")
|
||||
cmd = [
|
||||
"curl",
|
||||
"-X",
|
||||
"GET",
|
||||
f"{plex_url}/library/sections/1/refresh",
|
||||
"-H",
|
||||
f"X-Plex-Token: {plex_token}",
|
||||
]
|
||||
subprocess.run(cmd, capture_output=True, check=False)
|
||||
print(" ✓ Plex refresh triggered")
|
||||
except Exception as e:
|
||||
print(f" ⚠️ Failed to refresh Plex: {e}")
|
||||
|
||||
|
||||
def process_file(
|
||||
filepath,
|
||||
log_dir,
|
||||
log_name,
|
||||
preset,
|
||||
hw_encoder="svt-av1",
|
||||
use_hw_mode=False,
|
||||
hwaccel=None,
|
||||
):
|
||||
"""Process a single video file with intelligent VMAF targeting"""
|
||||
global _shutdown_requested
|
||||
|
||||
# --- LOGIC FIX START ---
|
||||
# If "use_hw_mode" (Hybrid) is FALSE, use the requested encoder for EVERYONE.
|
||||
if not use_hw_mode:
|
||||
encoder = hw_encoder
|
||||
use_hw = (encoder in HW_ENCODERS)
|
||||
else:
|
||||
# Hybrid Mode: Only 1 worker gets HW, rest get CPU
|
||||
use_hw = False
|
||||
if hwaccel and hw_encoder in HW_ENCODERS:
|
||||
use_hw = claim_hardware_worker()
|
||||
encoder = hw_encoder if use_hw else "svt-av1"
|
||||
# --- LOGIC FIX END ---
|
||||
|
||||
filepath = Path(filepath)
|
||||
lock_file = Path(log_dir).parent / ".lock" / f"{filepath.name}.lock"
|
||||
|
||||
# Check lock file (multi-machine coordination)
|
||||
if lock_file.exists():
|
||||
print(f"Skipping (Locked): {filepath.name}")
|
||||
return True
|
||||
|
||||
# Create lock
|
||||
lock_file.parent.mkdir(parents=True, exist_ok=True)
|
||||
lock_file.touch()
|
||||
|
||||
try:
|
||||
probe_data = get_probe_data(filepath)
|
||||
if not probe_data:
|
||||
print(f"Skipping (Invalid probe data): {filepath.name}")
|
||||
return True
|
||||
stats_before = get_video_stats(probe_data)
|
||||
|
||||
if not stats_before or stats_before["codec"] == "av1":
|
||||
print(f"Skipping (Already AV1 or invalid): {filepath.name}")
|
||||
return True
|
||||
|
||||
# Mark as processed
|
||||
file_key = str(filepath)
|
||||
with _lock:
|
||||
if file_key in _processed_files:
|
||||
return True
|
||||
_processed_files.add(file_key)
|
||||
|
||||
print(f"\n--- Processing: {filepath.name} ---")
|
||||
# Added 'errors="ignore"' to print to avoid Unicode crashes on console
|
||||
try:
|
||||
print(
|
||||
f" Source: {stats_before['codec']} @ {stats_before['bitrate']}k, {stats_before['size'] / (1024**3):.2f} GB"
|
||||
)
|
||||
except UnicodeEncodeError:
|
||||
print(f" Source: {stats_before['codec']} @ {stats_before['bitrate']}k")
|
||||
|
||||
if _shutdown_requested:
|
||||
return False
|
||||
|
||||
temp_dir = Path(log_dir).parent / "tmp"
|
||||
temp_dir.mkdir(exist_ok=True)
|
||||
|
||||
# Step 1: Try VMAF 94
|
||||
print(f"\n [Step 1] Testing VMAF 94...")
|
||||
search_result_94 = run_crf_search(
|
||||
filepath, 94.0, preset, str(temp_dir), encoder, use_hw, hwaccel
|
||||
)
|
||||
|
||||
if not search_result_94:
|
||||
print(f" !! Could not hit VMAF 94")
|
||||
search_result_94 = run_crf_search(
|
||||
filepath, 93.0, preset, str(temp_dir), encoder, use_hw, hwaccel
|
||||
)
|
||||
if not search_result_94:
|
||||
search_result_94 = run_crf_search(
|
||||
filepath, 92.0, preset, str(temp_dir), encoder, use_hw, hwaccel
|
||||
)
|
||||
if not search_result_94:
|
||||
search_result_94 = run_crf_search(
|
||||
filepath, 90.0, preset, str(temp_dir), encoder, use_hw, hwaccel
|
||||
)
|
||||
|
||||
if not search_result_94:
|
||||
print(f" !! Failed all VMAF targets ({TARGETS}) for {filepath.name}")
|
||||
log_result(
|
||||
log_dir,
|
||||
"failed_searches",
|
||||
{
|
||||
"file": str(filepath),
|
||||
"status": "all_targets_failed",
|
||||
"targets": TARGETS,
|
||||
},
|
||||
)
|
||||
provide_recommendations(stats_before, None, 0)
|
||||
return False
|
||||
|
||||
crf_94 = search_result_94["crf"]
|
||||
predicted_savings_94 = 100.0 - search_result_94["predicted_percent"]
|
||||
|
||||
if predicted_savings_94 >= MIN_SAVINGS_PERCENT:
|
||||
print(
|
||||
f"\n ✅ VMAF 94 gives {predicted_savings_94:.1f}% savings (≥{MIN_SAVINGS_PERCENT}%)"
|
||||
)
|
||||
print(f" → Proceeding with VMAF 94, CRF {crf_94}")
|
||||
encode_params = {
|
||||
"crf": crf_94,
|
||||
"vmaf": 94.0,
|
||||
"predicted_percent": search_result_94["predicted_percent"],
|
||||
}
|
||||
else:
|
||||
print(
|
||||
f"\n ⚠️ VMAF 94 gives {predicted_savings_94:.1f}% savings (<{MIN_SAVINGS_PERCENT}%)"
|
||||
)
|
||||
|
||||
search_result_93 = run_crf_search(
|
||||
filepath, 93.0, preset, str(temp_dir), encoder, use_hw, hwaccel
|
||||
)
|
||||
if search_result_93:
|
||||
predicted_savings_93 = 100.0 - search_result_93["predicted_percent"]
|
||||
|
||||
if predicted_savings_93 >= MIN_SAVINGS_PERCENT:
|
||||
print(
|
||||
f" ✅ VMAF 93 gives {predicted_savings_93:.1f}% savings (≥{MIN_SAVINGS_PERCENT}%)"
|
||||
)
|
||||
print(
|
||||
f" → Proceeding with VMAF 93, CRF {search_result_93['crf']}"
|
||||
)
|
||||
encode_params = {
|
||||
"crf": search_result_93["crf"],
|
||||
"vmaf": 93.0,
|
||||
"predicted_percent": search_result_93["predicted_percent"],
|
||||
}
|
||||
else:
|
||||
print(
|
||||
f" ⚠️ VMAF 93 gives {predicted_savings_93:.1f}% savings (also <{MIN_SAVINGS_PERCENT}%)"
|
||||
)
|
||||
print(
|
||||
f" → Finding VMAF for {TARGET_SAVINGS_FOR_ESTIMATE}% savings..."
|
||||
)
|
||||
target_result = find_target_savings_params(
|
||||
filepath, 93.0, preset, str(temp_dir), encoder, use_hw, hwaccel
|
||||
)
|
||||
|
||||
provide_recommendations(
|
||||
stats_before, 93.0, predicted_savings_93, target_result
|
||||
)
|
||||
log_result(
|
||||
log_dir,
|
||||
"low_savings_skips",
|
||||
{
|
||||
"file": str(filepath),
|
||||
"vmaf_94": 94.0,
|
||||
"savings_94": predicted_savings_94,
|
||||
"vmaf_93": 93.0,
|
||||
"savings_93": predicted_savings_93,
|
||||
"target_for_15_percent": target_result,
|
||||
"recommendations": "logged_for_review",
|
||||
},
|
||||
)
|
||||
return True
|
||||
else:
|
||||
print(f" !! Could not achieve VMAF 93")
|
||||
log_result(
|
||||
log_dir,
|
||||
"failed_searches",
|
||||
{"file": str(filepath), "status": "vmaf_93_failed"},
|
||||
)
|
||||
return False
|
||||
|
||||
temp_output = temp_dir / f"{filepath.stem}.av1_temp.mkv"
|
||||
if temp_output.exists():
|
||||
temp_output.unlink()
|
||||
|
||||
start_time = time.time()
|
||||
res = run_encode(
|
||||
filepath,
|
||||
temp_output,
|
||||
encode_params["crf"],
|
||||
preset,
|
||||
encoder,
|
||||
use_hw,
|
||||
hwaccel,
|
||||
)
|
||||
|
||||
if res != 0:
|
||||
print(f"\n !! Encode failed with code {res}")
|
||||
if temp_output.exists():
|
||||
temp_output.unlink()
|
||||
log_result(
|
||||
log_dir,
|
||||
"failed_encodes",
|
||||
{"file": str(filepath), "status": "encode_failed", "returncode": res},
|
||||
)
|
||||
return False
|
||||
|
||||
encode_duration = time.time() - start_time
|
||||
print(f" ✓ Encode completed in {encode_duration:.1f}s")
|
||||
|
||||
probe_after = get_probe_data(temp_output)
|
||||
stats_after = get_video_stats(probe_after)
|
||||
|
||||
if not stats_after:
|
||||
print(f" !! Failed to probe encoded file")
|
||||
if temp_output.exists():
|
||||
temp_output.unlink()
|
||||
return False
|
||||
|
||||
actual_savings = (1 - (stats_after["size"] / stats_before["size"])) * 100
|
||||
|
||||
print(f"\n Results:")
|
||||
print(
|
||||
f" - Before: {stats_before['size'] / (1024**3):.2f} GB @ {stats_before['bitrate']}k"
|
||||
)
|
||||
print(
|
||||
f" - After: {stats_after['size'] / (1024**3):.2f} GB @ {stats_after['bitrate']}k"
|
||||
)
|
||||
print(f" - Savings: {actual_savings:.2f}%")
|
||||
|
||||
final_path = filepath
|
||||
if filepath.suffix.lower() == ".mp4":
|
||||
final_path = filepath.with_suffix(".mkv")
|
||||
if final_path.exists():
|
||||
final_path.unlink()
|
||||
shutil.move(str(filepath), str(final_path))
|
||||
|
||||
shutil.move(str(temp_output), str(final_path))
|
||||
|
||||
print(f" ✓ Successfully optimized: {final_path.name}")
|
||||
|
||||
log_result(
|
||||
log_dir,
|
||||
log_name,
|
||||
{
|
||||
"file": str(final_path),
|
||||
"status": "success",
|
||||
"vmaf": encode_params["vmaf"],
|
||||
"crf": encode_params["crf"],
|
||||
"before": stats_before,
|
||||
"after": stats_after,
|
||||
"duration": encode_duration,
|
||||
"savings": actual_savings,
|
||||
},
|
||||
)
|
||||
return True
|
||||
|
||||
finally:
|
||||
# Remove lock file
|
||||
if lock_file.exists():
|
||||
lock_file.unlink()
|
||||
|
||||
|
||||
def scan_library(root, exclude_dirs=None):
|
||||
"""Scan library for video files, excluding certain directories"""
|
||||
exclude_dirs = exclude_dirs or []
|
||||
|
||||
files = []
|
||||
for dirpath, dirnames, filenames in os.walk(root):
|
||||
# Skip excluded directories
|
||||
dirnames[:] = [d for d in dirnames if d not in exclude_dirs]
|
||||
|
||||
for filename in filenames:
|
||||
if Path(filename).suffix.lower() not in EXTENSIONS:
|
||||
continue
|
||||
|
||||
full_path = Path(dirpath) / filename
|
||||
if "_av1" in full_path.stem:
|
||||
continue
|
||||
|
||||
files.append(full_path)
|
||||
|
||||
return files
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Optimize video library to AV1 using VMAF targeting."
|
||||
)
|
||||
parser.add_argument("directory", help="Root directory to scan")
|
||||
parser.add_argument(
|
||||
"--vmaf",
|
||||
type=float,
|
||||
default=DEFAULT_VMAF,
|
||||
help=f"Target VMAF score (default: {DEFAULT_VMAF})",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--preset",
|
||||
type=int,
|
||||
default=DEFAULT_PRESET,
|
||||
help=f"SVT-AV1 Preset (default: {DEFAULT_PRESET})",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--workers",
|
||||
type=int,
|
||||
default=DEFAULT_WORKERS,
|
||||
help=f"Concurrent files to process (default: {DEFAULT_WORKERS})",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--samples",
|
||||
type=int,
|
||||
default=DEFAULT_SAMPLES,
|
||||
help=f"Samples to use for CRF search if supported (default: {DEFAULT_SAMPLES})",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--hwaccel",
|
||||
default=None,
|
||||
help=(
|
||||
"Hardware acceleration for decode. "
|
||||
"Examples: auto, vaapi, d3d11va, videotoolbox. Use 'none' to disable."
|
||||
),
|
||||
)
|
||||
parser.add_argument(
|
||||
"--encoder",
|
||||
default="svt-av1",
|
||||
help=(
|
||||
"Video encoder to use. Default: svt-av1 (CPU). "
|
||||
"Hardware encoders: av1_amf (AMD), av1_nvenc (NVIDIA), av1_qsv (Intel)."
|
||||
),
|
||||
)
|
||||
parser.add_argument(
|
||||
"--use-hardware-worker",
|
||||
action="store_true",
|
||||
help="Use 1 hardware encoding worker + rest CPU workers (requires --encoder with HW encoder)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--plex-url",
|
||||
default=None,
|
||||
help="Plex server URL (e.g., http://localhost:32400)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--plex-token",
|
||||
default=None,
|
||||
help="Plex auth token",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--log-dir",
|
||||
default="/opt/Optmiser/logs",
|
||||
help="Log directory (default: /opt/Optmiser/logs)",
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.workers < 1:
|
||||
print("Error: --workers must be >= 1")
|
||||
sys.exit(2)
|
||||
|
||||
check_dependencies()
|
||||
|
||||
root = Path(args.directory)
|
||||
if not root.exists():
|
||||
print(f"Directory not found: {root}")
|
||||
sys.exit(1)
|
||||
|
||||
hwaccel = normalize_hwaccel(args.hwaccel)
|
||||
|
||||
print(f"Platform: {platform_label()}")
|
||||
print(f"Scanning library: {root}")
|
||||
print(f"VMAF targets: {TARGETS}")
|
||||
print(f"Minimum savings: {MIN_SAVINGS_PERCENT}%")
|
||||
print(f"Estimate target: {TARGET_SAVINGS_FOR_ESTIMATE}%")
|
||||
print(f"Encoder Preset: {args.preset}")
|
||||
print(f"Workers: {args.workers}")
|
||||
if hwaccel:
|
||||
print(f"HWAccel: {hwaccel}")
|
||||
if args.use_hardware_worker:
|
||||
print(f"Hardware worker: 1 HW + {args.workers - 1} CPU")
|
||||
if args.plex_url:
|
||||
print(f"Plex refresh: Enabled")
|
||||
print("-" * 60)
|
||||
|
||||
# Determine log name based on directory
|
||||
root_parts = str(root).lower().split("/")
|
||||
if "content" in root_parts:
|
||||
log_name = "content"
|
||||
exclude_dirs = []
|
||||
else:
|
||||
log_name = "tv_movies"
|
||||
exclude_dirs = ["content"]
|
||||
|
||||
print(f"Log file: {log_name}.jsonl")
|
||||
|
||||
files = scan_library(root, exclude_dirs)
|
||||
if not files:
|
||||
print("No media files found.")
|
||||
return
|
||||
|
||||
print(f"Found {len(files)} files to process")
|
||||
print("-" * 60)
|
||||
|
||||
processed_count = 0
|
||||
success_count = 0
|
||||
fail_count = 0
|
||||
|
||||
# Hardware worker configuration
|
||||
# HW worker uses specified encoder; CPU workers use svt-av1
|
||||
hw_encoder = args.encoder if args.encoder in HW_ENCODERS else None
|
||||
use_hw_primary = args.use_hardware_worker and hw_encoder is not None
|
||||
|
||||
if args.workers == 1:
|
||||
# Single thread - just process files
|
||||
for file_path in files:
|
||||
if _shutdown_requested:
|
||||
break
|
||||
processed_count += 1
|
||||
result = process_file(
|
||||
file_path,
|
||||
args.log_dir,
|
||||
log_name,
|
||||
args.preset,
|
||||
args.encoder,
|
||||
use_hw_primary,
|
||||
hwaccel,
|
||||
)
|
||||
if result:
|
||||
success_count += 1
|
||||
else:
|
||||
fail_count += 1
|
||||
else:
|
||||
# Multi-threaded processing
|
||||
with ThreadPoolExecutor(max_workers=args.workers) as executor:
|
||||
futures = []
|
||||
for file_path in files:
|
||||
if _shutdown_requested:
|
||||
break
|
||||
|
||||
# All workers try to claim HW; only the first thread succeeds
|
||||
# and will use HW for ALL its tasks
|
||||
future = executor.submit(
|
||||
process_file,
|
||||
file_path,
|
||||
args.log_dir,
|
||||
log_name,
|
||||
args.preset,
|
||||
args.encoder,
|
||||
use_hw_primary,
|
||||
hwaccel,
|
||||
)
|
||||
futures.append(future)
|
||||
|
||||
for future in as_completed(futures):
|
||||
if _shutdown_requested:
|
||||
break
|
||||
|
||||
processed_count += 1
|
||||
try:
|
||||
result = future.result()
|
||||
if result:
|
||||
success_count += 1
|
||||
else:
|
||||
fail_count += 1
|
||||
except Exception as e:
|
||||
print(f" !! Unexpected error: {e}")
|
||||
import traceback
|
||||
|
||||
traceback.print_exc()
|
||||
fail_count += 1
|
||||
|
||||
print("\n" + "=" * 60)
|
||||
print(f"SUMMARY: {root}")
|
||||
print(f" Processed: {processed_count} files")
|
||||
print(f" Success/Skip: {success_count}")
|
||||
print(f" Failed: {fail_count}")
|
||||
print("=" * 60)
|
||||
|
||||
# Refresh Plex on completion
|
||||
if success_count > 0:
|
||||
refresh_plex(args.plex_url, args.plex_token)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -5,16 +5,18 @@ param(
|
||||
[int]$Preset = 6,
|
||||
[int]$Workers = 1,
|
||||
[int]$Samples = 4,
|
||||
[switch]$Thorough,
|
||||
[string]$Hwaccel = "",
|
||||
[string]$Encoder = "svt-av1",
|
||||
[string]$Hwaccel
|
||||
[switch]$UseHardwareWorker,
|
||||
[string]$PlexUrl = "",
|
||||
[string]$PlexToken = "",
|
||||
[string]$LogDir = "/opt/Optmiser/logs"
|
||||
)
|
||||
|
||||
$ErrorActionPreference = "Stop"
|
||||
|
||||
function Write-ColorOutput {
|
||||
param([string]$Message, [string]$Color = "White")
|
||||
|
||||
Write-Host $Message -ForegroundColor $Color
|
||||
}
|
||||
|
||||
@@ -26,7 +28,7 @@ function Invoke-OptimizeLibrary {
|
||||
exit 1
|
||||
}
|
||||
|
||||
$pythonCmd = Get-Command python3, python, py | Select-Object -FirstProperty Path -ErrorAction SilentlyContinue
|
||||
$pythonCmd = Get-Command "python"
|
||||
if (-not $pythonCmd) {
|
||||
Write-ColorOutput -Message "ERROR: Python 3 not found. Please install Python 3." -Color "Red"
|
||||
exit 1
|
||||
@@ -38,18 +40,27 @@ function Invoke-OptimizeLibrary {
|
||||
"--vmaf", $Vmaf.ToString("F1"),
|
||||
"--preset", $Preset.ToString(),
|
||||
"--workers", $Workers.ToString(),
|
||||
"--samples", $Samples.ToString()
|
||||
"--encoder", $Encoder
|
||||
"--samples", $Samples.ToString(),
|
||||
"--encoder", $Encoder,
|
||||
"--log-dir", $LogDir
|
||||
)
|
||||
|
||||
if ($Thorough) {
|
||||
$arguments += "--thorough"
|
||||
}
|
||||
|
||||
if ($Hwaccel) {
|
||||
$arguments += "--hwaccel", $Hwaccel
|
||||
}
|
||||
|
||||
if ($UseHardwareWorker) {
|
||||
$arguments += "--use-hardware-worker"
|
||||
}
|
||||
|
||||
if ($PlexUrl) {
|
||||
$arguments += "--plex-url", $PlexUrl
|
||||
}
|
||||
|
||||
if ($PlexToken) {
|
||||
$arguments += "--plex-token", $PlexToken
|
||||
}
|
||||
|
||||
Write-ColorOutput -Message "Running optimize_library.py..." -Color "Cyan"
|
||||
Write-ColorOutput -Message " Directory: $Directory" -Color "White"
|
||||
Write-ColorOutput -Message " Target VMAF: $Vmaf" -Color "White"
|
||||
@@ -57,16 +68,18 @@ function Invoke-OptimizeLibrary {
|
||||
Write-ColorOutput -Message " Workers: $Workers" -Color "White"
|
||||
Write-ColorOutput -Message " Samples: $Samples" -Color "White"
|
||||
Write-ColorOutput -Message " Encoder: $Encoder" -Color "White"
|
||||
if ($Thorough) {
|
||||
Write-ColorOutput -Message " Thorough: Yes" -Color "White"
|
||||
}
|
||||
if ($Hwaccel) {
|
||||
Write-ColorOutput -Message " HW Accel: $Hwaccel" -Color "White"
|
||||
}
|
||||
if ($UseHardwareWorker) {
|
||||
Write-ColorOutput -Message " Hardware worker: Enabled (1 HW + $($Workers - 1) CPU)" -Color "White"
|
||||
}
|
||||
if ($PlexUrl -and $PlexToken) {
|
||||
Write-ColorOutput -Message " Plex refresh: Enabled" -Color "White"
|
||||
}
|
||||
Write-Host ""
|
||||
|
||||
$process = Start-Process -FilePath $pythonCmd.Path -ArgumentList $arguments -NoNewWindow -PassThru
|
||||
|
||||
$process.WaitForExit()
|
||||
$exitCode = $process.ExitCode
|
||||
|
||||
@@ -1,8 +1,5 @@
|
||||
#!/bin/bash
|
||||
|
||||
# VMAF Library Optimiser (Linux/Server runner)
|
||||
# This script wraps optimize_library.py with the same interface as the Windows PowerShell version
|
||||
|
||||
set -e
|
||||
|
||||
COLOR_RED='\033[0;31m'
|
||||
@@ -24,15 +21,17 @@ log_success() {
|
||||
echo -e "${COLOR_GREEN}$*${COLOR_RESET}"
|
||||
}
|
||||
|
||||
# Default values matching optimize_library.py defaults
|
||||
DIRECTORY="."
|
||||
VMAF="95.0"
|
||||
PRESET="6"
|
||||
WORKERS="1"
|
||||
SAMPLES="4"
|
||||
THOROUGH=""
|
||||
ENCODER="svt-av1"
|
||||
HWACCEL=""
|
||||
USE_HW_WORKER=""
|
||||
PLEX_URL=""
|
||||
PLEX_TOKEN=""
|
||||
LOG_DIR="/opt/Optmiser/logs"
|
||||
|
||||
|
||||
# Parse command line arguments
|
||||
while [[ $# -gt 0 ]]; do
|
||||
@@ -57,18 +56,26 @@ while [[ $# -gt 0 ]]; do
|
||||
SAMPLES="$2"
|
||||
shift 2
|
||||
;;
|
||||
--thorough)
|
||||
THOROUGH="--thorough"
|
||||
shift
|
||||
;;
|
||||
--encoder)
|
||||
ENCODER="$2"
|
||||
shift 2
|
||||
;;
|
||||
--hwaccel)
|
||||
HWACCEL="$2"
|
||||
shift 2
|
||||
;;
|
||||
--use-hardware-worker)
|
||||
USE_HARDWARE_WORKER="true"
|
||||
shift
|
||||
;;
|
||||
--plex-url)
|
||||
PLEX_URL="$2"
|
||||
shift 2
|
||||
;;
|
||||
--plex-token)
|
||||
PLEX_TOKEN="$2"
|
||||
shift 2
|
||||
;;
|
||||
--log-dir)
|
||||
LOG_DIR="$2"
|
||||
shift 2
|
||||
;;
|
||||
*)
|
||||
DIRECTORY="$1"
|
||||
shift
|
||||
@@ -105,7 +112,7 @@ ARGS=(
|
||||
--preset "$PRESET"
|
||||
--workers "$WORKERS"
|
||||
--samples "$SAMPLES"
|
||||
--encoder "$ENCODER"
|
||||
--log-dir "$LOG_DIR"
|
||||
)
|
||||
|
||||
if [[ -n "$THOROUGH" ]]; then
|
||||
30
legacy/run_smart_encoder.ps1
Normal file
30
legacy/run_smart_encoder.ps1
Normal file
@@ -0,0 +1,30 @@
|
||||
# Run Smart GPU Encoder
|
||||
# Wrapper to ensure correct environment
|
||||
|
||||
$ScriptPath = "$PSScriptRoot\smart_gpu_encoder.py"
|
||||
$PythonPath = "python" # Or specific path like "C:\Python39\python.exe"
|
||||
|
||||
# Directories (Change these to your actual paths)
|
||||
$TvDir = "Z:\tv"
|
||||
$ContentDir = "Z:\content"
|
||||
|
||||
Write-Host "==========================================" -ForegroundColor Cyan
|
||||
Write-Host " SMART GPU ENCODER (AMD AMF + VMAF)" -ForegroundColor Cyan
|
||||
Write-Host "==========================================" -ForegroundColor Cyan
|
||||
Write-Host "Script: $ScriptPath"
|
||||
Write-Host "TV Dir: $TvDir"
|
||||
Write-Host "Content Dir: $ContentDir"
|
||||
Write-Host ""
|
||||
|
||||
# Check if ab-av1 exists in bin
|
||||
if (-not (Test-Path "$PSScriptRoot\bin\ab-av1.exe")) {
|
||||
Write-Host "WARNING: ab-av1.exe not found in bin folder!" -ForegroundColor Red
|
||||
Write-Host "Please download it and place it in $PSScriptRoot\bin\"
|
||||
exit 1
|
||||
}
|
||||
|
||||
# Run the python script
|
||||
& $PythonPath $ScriptPath --tv-dir $TvDir --content-dir $ContentDir
|
||||
|
||||
Write-Host "`nDone." -ForegroundColor Green
|
||||
Read-Host "Press Enter to exit..."
|
||||
@@ -1,341 +0,0 @@
|
||||
import os
|
||||
import sys
|
||||
import subprocess
|
||||
import argparse
|
||||
import json
|
||||
import shutil
|
||||
import platform
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
from pathlib import Path
|
||||
|
||||
DEFAULT_VMAF = 95.0
|
||||
DEFAULT_PRESET = 6
|
||||
DEFAULT_WORKERS = 1
|
||||
DEFAULT_SAMPLES = 4
|
||||
EXTENSIONS = {".mkv", ".mp4", ".mov", ".avi", ".ts"}
|
||||
|
||||
_AB_AV1_HELP_CACHE = {}
|
||||
|
||||
|
||||
def check_dependencies():
|
||||
missing = []
|
||||
for tool in ["ffmpeg", "ffprobe", "ab-av1"]:
|
||||
if not shutil.which(tool):
|
||||
missing.append(tool)
|
||||
|
||||
if missing:
|
||||
print(f"Error: Missing required tools: {', '.join(missing)}")
|
||||
print(
|
||||
"Please install FFmpeg and 'ab-av1' (via cargo install ab-av1) before running."
|
||||
)
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
def is_wsl():
|
||||
if os.environ.get("WSL_DISTRO_NAME"):
|
||||
return True
|
||||
|
||||
try:
|
||||
with open("/proc/sys/kernel/osrelease", "r", encoding="utf-8") as f:
|
||||
return "microsoft" in f.read().lower()
|
||||
except FileNotFoundError:
|
||||
return False
|
||||
|
||||
|
||||
def platform_label():
|
||||
system = platform.system()
|
||||
if system == "Linux" and is_wsl():
|
||||
return "Linux (WSL)"
|
||||
return system
|
||||
|
||||
|
||||
def _ab_av1_help(subcommand):
|
||||
cached = _AB_AV1_HELP_CACHE.get(subcommand)
|
||||
if cached is not None:
|
||||
return cached
|
||||
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["ab-av1", subcommand, "--help"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=False,
|
||||
)
|
||||
help_text = (result.stdout or "") + "\n" + (result.stderr or "")
|
||||
except Exception:
|
||||
help_text = ""
|
||||
|
||||
_AB_AV1_HELP_CACHE[subcommand] = help_text
|
||||
return help_text
|
||||
|
||||
|
||||
def ab_av1_supports(subcommand, flag):
|
||||
return flag in _ab_av1_help(subcommand)
|
||||
|
||||
|
||||
def normalize_hwaccel(value):
|
||||
if value is None:
|
||||
return None
|
||||
|
||||
v = value.strip()
|
||||
if not v:
|
||||
return None
|
||||
|
||||
v_lower = v.lower()
|
||||
if v_lower in {"none", "off", "false", "0"}:
|
||||
return None
|
||||
|
||||
if v_lower != "auto":
|
||||
return v
|
||||
|
||||
system = platform.system()
|
||||
if system == "Windows":
|
||||
return "d3d11va"
|
||||
if system == "Darwin":
|
||||
return "videotoolbox"
|
||||
|
||||
return "vaapi"
|
||||
|
||||
|
||||
def get_video_info(filepath):
|
||||
try:
|
||||
cmd = [
|
||||
"ffprobe",
|
||||
"-v",
|
||||
"quiet",
|
||||
"-print_format",
|
||||
"json",
|
||||
"-show_streams",
|
||||
"-select_streams",
|
||||
"v:0",
|
||||
filepath,
|
||||
]
|
||||
result = subprocess.run(cmd, capture_output=True, text=True, check=True)
|
||||
data = json.loads(result.stdout)
|
||||
|
||||
streams = data.get("streams") or []
|
||||
if not streams:
|
||||
return None
|
||||
|
||||
stream = streams[0]
|
||||
codec = stream.get("codec_name", "unknown")
|
||||
color_transfer = stream.get("color_transfer", "unknown")
|
||||
is_hdr = color_transfer in ["smpte2084", "arib-std-b67"]
|
||||
|
||||
return {"codec": codec, "is_hdr": is_hdr}
|
||||
except Exception as e:
|
||||
print(f"Error probing {filepath}: {e}")
|
||||
return None
|
||||
|
||||
|
||||
def build_ab_av1_command(input_path, output_path, args):
|
||||
cmd = [
|
||||
"ab-av1",
|
||||
"auto-encode",
|
||||
"-i",
|
||||
str(input_path),
|
||||
"-o",
|
||||
str(output_path),
|
||||
"--min-vmaf",
|
||||
str(args.vmaf),
|
||||
"--preset",
|
||||
str(args.preset),
|
||||
]
|
||||
|
||||
if args.encoder:
|
||||
if ab_av1_supports("auto-encode", "--encoder"):
|
||||
cmd.extend(["--encoder", args.encoder])
|
||||
elif ab_av1_supports("auto-encode", "-e"):
|
||||
cmd.extend(["-e", args.encoder])
|
||||
else:
|
||||
print("Warning: This ab-av1 version does not support --encoder; ignoring.")
|
||||
|
||||
if args.samples is not None:
|
||||
if ab_av1_supports("auto-encode", "--samples"):
|
||||
cmd.extend(["--samples", str(args.samples)])
|
||||
elif ab_av1_supports("auto-encode", "--sample-count"):
|
||||
cmd.extend(["--sample-count", str(args.samples)])
|
||||
else:
|
||||
print("Warning: This ab-av1 version does not support --samples; ignoring.")
|
||||
|
||||
if args.thorough:
|
||||
if ab_av1_supports("auto-encode", "--thorough"):
|
||||
cmd.append("--thorough")
|
||||
else:
|
||||
print("Warning: This ab-av1 version does not support --thorough; ignoring.")
|
||||
|
||||
hwaccel = normalize_hwaccel(args.hwaccel)
|
||||
if hwaccel is not None:
|
||||
if ab_av1_supports("auto-encode", "--enc-input"):
|
||||
cmd.extend(["--enc-input", f"hwaccel={hwaccel}"])
|
||||
hwaccel_output_format = args.hwaccel_output_format
|
||||
if hwaccel_output_format is None and hwaccel == "vaapi":
|
||||
hwaccel_output_format = "vaapi"
|
||||
if hwaccel_output_format is not None:
|
||||
cmd.extend(
|
||||
["--enc-input", f"hwaccel_output_format={hwaccel_output_format}"]
|
||||
)
|
||||
else:
|
||||
print(
|
||||
"Warning: This ab-av1 version does not support --enc-input; ignoring --hwaccel."
|
||||
)
|
||||
|
||||
if ab_av1_supports("auto-encode", "--acodec"):
|
||||
cmd.extend(["--acodec", "copy"])
|
||||
elif ab_av1_supports("auto-encode", "--ac"):
|
||||
cmd.extend(["--ac", "copy"])
|
||||
else:
|
||||
print(
|
||||
"Warning: This ab-av1 version does not support --acodec/--ac; leaving audio defaults."
|
||||
)
|
||||
|
||||
return cmd
|
||||
|
||||
|
||||
def process_file(filepath, args):
|
||||
input_path = Path(filepath)
|
||||
output_path = input_path.with_stem(input_path.stem + "_av1")
|
||||
|
||||
if output_path.exists():
|
||||
print(f"Skipping (Output exists): {input_path.name}")
|
||||
return
|
||||
|
||||
info = get_video_info(str(input_path))
|
||||
if not info:
|
||||
return
|
||||
|
||||
if info["codec"] == "av1":
|
||||
print(f"Skipping (Already AV1): {input_path.name}")
|
||||
return
|
||||
|
||||
print(f"\nProcessing: {input_path.name}")
|
||||
print(f" Source Codec: {info['codec']}")
|
||||
print(f" HDR: {info['is_hdr']}")
|
||||
|
||||
cmd = build_ab_av1_command(input_path, output_path, args)
|
||||
|
||||
try:
|
||||
subprocess.run(cmd, check=True)
|
||||
print(f"Success! Encoded: {output_path.name}")
|
||||
except subprocess.CalledProcessError:
|
||||
print(f"Failed to encode: {input_path.name}")
|
||||
if output_path.exists():
|
||||
os.remove(output_path)
|
||||
|
||||
|
||||
def scan_library(root):
|
||||
files = []
|
||||
for dirpath, _, filenames in os.walk(root):
|
||||
for filename in filenames:
|
||||
if Path(filename).suffix.lower() not in EXTENSIONS:
|
||||
continue
|
||||
|
||||
full_path = Path(dirpath) / filename
|
||||
if "_av1" in full_path.stem:
|
||||
continue
|
||||
|
||||
files.append(full_path)
|
||||
|
||||
return files
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Optimize video library to AV1 using VMAF targeting."
|
||||
)
|
||||
parser.add_argument("directory", help="Root directory to scan")
|
||||
parser.add_argument(
|
||||
"--vmaf",
|
||||
type=float,
|
||||
default=DEFAULT_VMAF,
|
||||
help=f"Target VMAF score (default: {DEFAULT_VMAF})",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--preset",
|
||||
type=int,
|
||||
default=DEFAULT_PRESET,
|
||||
help=f"SVT-AV1 Preset (default: {DEFAULT_PRESET})",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--workers",
|
||||
type=int,
|
||||
default=DEFAULT_WORKERS,
|
||||
help=f"Concurrent files to process (default: {DEFAULT_WORKERS})",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--samples",
|
||||
type=int,
|
||||
default=DEFAULT_SAMPLES,
|
||||
help=f"Samples to use for CRF search if supported (default: {DEFAULT_SAMPLES})",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--thorough",
|
||||
action="store_true",
|
||||
help="Use ab-av1 thorough mode if supported (slower, more accurate)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--encoder",
|
||||
default="svt-av1",
|
||||
help="ab-av1 encoder (default: svt-av1). For AMD AV1 on Windows try: av1_amf",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--hwaccel",
|
||||
default=None,
|
||||
help=(
|
||||
"Hardware acceleration for decode (passed via ab-av1 --enc-input if supported). "
|
||||
"Examples: auto, vaapi, d3d11va, videotoolbox. Use 'none' to disable."
|
||||
),
|
||||
)
|
||||
parser.add_argument(
|
||||
"--hwaccel-output-format",
|
||||
default=None,
|
||||
help="Optional hwaccel_output_format override (e.g., vaapi)",
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.workers < 1:
|
||||
print("Error: --workers must be >= 1")
|
||||
sys.exit(2)
|
||||
|
||||
check_dependencies()
|
||||
|
||||
root = Path(args.directory)
|
||||
if not root.exists():
|
||||
print(f"Directory not found: {root}")
|
||||
sys.exit(1)
|
||||
|
||||
print(f"Platform: {platform_label()}")
|
||||
print(f"Scanning library: {root}")
|
||||
print(f"Target VMAF: {args.vmaf}")
|
||||
print(f"Encoder Preset: {args.preset}")
|
||||
print(f"Workers: {args.workers}")
|
||||
print(f"Samples: {args.samples}")
|
||||
print(f"Encoder: {args.encoder}")
|
||||
if args.hwaccel is not None:
|
||||
print(f"HWAccel: {args.hwaccel}")
|
||||
print("-" * 50)
|
||||
|
||||
files = scan_library(root)
|
||||
if not files:
|
||||
print("No media files found.")
|
||||
return
|
||||
|
||||
if args.workers == 1:
|
||||
for file_path in files:
|
||||
process_file(file_path, args)
|
||||
return
|
||||
|
||||
with ThreadPoolExecutor(max_workers=args.workers) as executor:
|
||||
futures = [
|
||||
executor.submit(process_file, file_path, args) for file_path in files
|
||||
]
|
||||
for future in as_completed(futures):
|
||||
try:
|
||||
future.result()
|
||||
except Exception as e:
|
||||
print(f"Unexpected error: {e}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
12
run.ps1
Normal file
12
run.ps1
Normal file
@@ -0,0 +1,12 @@
|
||||
# Run Smart GPU Encoder (Wrapper)
|
||||
$ScriptPath = "$PSScriptRoot\smart_gpu_encoder.py"
|
||||
$PythonPath = "python"
|
||||
|
||||
# Default to interactive mode if no args
|
||||
if ($args.Count -eq 0) {
|
||||
Write-Host "Starting Smart Encoder..." -ForegroundColor Cyan
|
||||
& $PythonPath $ScriptPath
|
||||
} else {
|
||||
# Pass through arguments
|
||||
& $PythonPath $ScriptPath $args
|
||||
}
|
||||
191
run_smart_encoder.sh
Normal file
191
run_smart_encoder.sh
Normal file
@@ -0,0 +1,191 @@
|
||||
#!/bin/bash
|
||||
# Smart Video Encoder Launcher - Linux/WSL
|
||||
# Usage: ./run_smart_encoder.sh [options]
|
||||
|
||||
set -e
|
||||
|
||||
# Colors
|
||||
RED='\033[0;31m'
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
CYAN='\033[0;36m'
|
||||
MAGENTA='\033[0;35m'
|
||||
RESET='\033[0m'
|
||||
|
||||
# Configuration
|
||||
TV_DIR="${TV_DIR:-/mnt/z/tv}"
|
||||
CONTENT_DIR="${CONTENT_DIR:-/mnt/z/content}"
|
||||
JOBS="${JOBS:-1}"
|
||||
TV_ONLY=false
|
||||
CONTENT_ONLY=false
|
||||
|
||||
# Parse arguments
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--tv-dir)
|
||||
TV_DIR="$2"
|
||||
shift 2
|
||||
;;
|
||||
--content-dir)
|
||||
CONTENT_DIR="$2"
|
||||
shift 2
|
||||
;;
|
||||
--jobs)
|
||||
JOBS="$2"
|
||||
shift 2
|
||||
;;
|
||||
--tv-only)
|
||||
TV_ONLY=true
|
||||
shift
|
||||
;;
|
||||
--content-only)
|
||||
CONTENT_ONLY=true
|
||||
shift
|
||||
;;
|
||||
--help|-h)
|
||||
echo "Usage: $0 [options]"
|
||||
echo ""
|
||||
echo "Options:"
|
||||
echo " --tv-dir <path> TV directory (default: /mnt/z/tv)"
|
||||
echo " --content-dir <path> Content directory (default: /mnt/z/content)"
|
||||
echo " --jobs <count> Parallel jobs (default: 1)"
|
||||
echo " --tv-only Process TV directory only"
|
||||
echo " --content-only Process content directory only"
|
||||
echo " --help, -h Show this help"
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
echo "Unknown option: $1"
|
||||
echo "Use --help for usage"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
log_info() {
|
||||
echo -e "${CYAN}$*${RESET}"
|
||||
}
|
||||
|
||||
log_success() {
|
||||
echo -e "${GREEN}✓ $*${RESET}"
|
||||
}
|
||||
|
||||
log_error() {
|
||||
echo -e "${RED}❌ $*${RESET}"
|
||||
}
|
||||
|
||||
log_warn() {
|
||||
echo -e "${YELLOW}⚠️ $*${RESET}"
|
||||
}
|
||||
|
||||
# Get script directory
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
PYTHON_SCRIPT="$SCRIPT_DIR/smart_encoder.py"
|
||||
|
||||
# Check dependencies
|
||||
log_info "Checking dependencies..."
|
||||
|
||||
if ! command -v python3 &> /dev/null; then
|
||||
log_error "Python 3 not found"
|
||||
log_warn "Install with: sudo apt install python3"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if ! command -v ffmpeg &> /dev/null; then
|
||||
log_error "FFmpeg not found"
|
||||
log_warn "Install with: sudo apt install ffmpeg"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if ! command -v ffprobe &> /dev/null; then
|
||||
log_error "FFprobe not found"
|
||||
log_warn "Install with: sudo apt install ffmpeg"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if ! command -v ab-av1 &> /dev/null; then
|
||||
log_error "ab-av1 not found"
|
||||
log_warn "Install with: cargo install ab-av1"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
log_success "All dependencies found"
|
||||
|
||||
# Check if script exists
|
||||
if [[ ! -f "$PYTHON_SCRIPT" ]]; then
|
||||
log_error "smart_encoder.py not found at: $PYTHON_SCRIPT"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Make script executable
|
||||
chmod +x "$PYTHON_SCRIPT"
|
||||
|
||||
# Print banner
|
||||
echo ""
|
||||
echo -e "${MAGENTA}============================================================${RESET}"
|
||||
echo -e "${MAGENTA}🎬 Smart Video Encoder - Linux/WSL${RESET}"
|
||||
echo -e "${MAGENTA}============================================================${RESET}"
|
||||
echo ""
|
||||
|
||||
# Print configuration
|
||||
log_info "Configuration:"
|
||||
echo " TV Directory: $TV_DIR"
|
||||
echo " Content Directory: $CONTENT_DIR"
|
||||
echo " Parallel Jobs: $JOBS"
|
||||
|
||||
if [[ "$TV_ONLY" == true ]]; then
|
||||
echo -e " ${YELLOW}Mode: TV only${RESET}"
|
||||
elif [[ "$CONTENT_ONLY" == true ]]; then
|
||||
echo -e " ${YELLOW}Mode: Content only${RESET}"
|
||||
else
|
||||
echo -e " ${YELLOW}Mode: TV + Content${RESET}"
|
||||
fi
|
||||
|
||||
# Check directories
|
||||
if [[ "$TV_ONLY" == false ]] && [[ ! -d "$TV_DIR" ]]; then
|
||||
log_warn "TV directory not found: $TV_DIR"
|
||||
fi
|
||||
|
||||
if [[ "$CONTENT_ONLY" == false ]] && [[ ! -d "$CONTENT_DIR" ]]; then
|
||||
log_warn "Content directory not found: $CONTENT_DIR"
|
||||
fi
|
||||
|
||||
# Build command
|
||||
CMD="python3 $PYTHON_SCRIPT --tv-dir $TV_DIR --content-dir $CONTENT_DIR --jobs $JOBS"
|
||||
|
||||
if [[ "$TV_ONLY" == true ]]; then
|
||||
CMD="$CMD --tv-only"
|
||||
elif [[ "$CONTENT_ONLY" == true ]]; then
|
||||
CMD="$CMD --content-only"
|
||||
fi
|
||||
|
||||
# Run
|
||||
echo ""
|
||||
log_info "Starting encoder..."
|
||||
echo -e "${CYAN}============================================================${RESET}"
|
||||
echo ""
|
||||
|
||||
eval $CMD
|
||||
EXIT_CODE=$?
|
||||
|
||||
echo ""
|
||||
echo -e "${CYAN}============================================================${RESET}"
|
||||
|
||||
if [[ $EXIT_CODE -eq 0 ]]; then
|
||||
log_success "Encoding completed successfully"
|
||||
else
|
||||
log_error "Encoding failed with exit code: $EXIT_CODE"
|
||||
fi
|
||||
|
||||
echo -e "${CYAN}============================================================${RESET}"
|
||||
|
||||
# Show log locations
|
||||
echo ""
|
||||
log_info "Log files:"
|
||||
echo " $HOME/Videos/encodes/logs/tv.jsonl"
|
||||
echo " $HOME/Videos/encodes/logs/content.jsonl"
|
||||
echo " $HOME/Videos/encodes/logs/rejected.jsonl"
|
||||
echo " $HOME/Videos/encodes/logs/metadata.jsonl"
|
||||
echo ""
|
||||
|
||||
exit $EXIT_CODE
|
||||
444
src/smart_encoder.py
Normal file
444
src/smart_encoder.py
Normal file
@@ -0,0 +1,444 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Smart Video Encoder - AV1 with HEVC Fallback
|
||||
Runs on Windows with native ab-av1.exe and ffmpeg
|
||||
Refactored to use vmaf_common.py
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
import subprocess
|
||||
import json
|
||||
import shutil
|
||||
import time
|
||||
import argparse
|
||||
import signal
|
||||
import platform
|
||||
import threading
|
||||
from pathlib import Path
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
|
||||
import vmaf_common as common
|
||||
|
||||
# --- Configuration ---
|
||||
TARGET_VMAF_MIN = common.DEFAULT_CONFIG["target_vmaf"]
|
||||
MIN_SAVINGS_PERCENT = common.DEFAULT_CONFIG["min_savings"]
|
||||
MAX_JOBS = common.DEFAULT_CONFIG["cpu_jobs"]
|
||||
|
||||
AV1_QUALITY = common.DEFAULT_CONFIG["av1_crf"]
|
||||
HEVC_QUALITY = common.DEFAULT_CONFIG["hevc_crf"]
|
||||
|
||||
SAMPLE_DURATION = 60
|
||||
SAMPLE_START = 300
|
||||
MAX_SAMPLE_SIZE_MB = 150
|
||||
|
||||
TEMP_DIR = common.get_temp_dir()
|
||||
|
||||
# Tools
|
||||
FFMPEG_BIN = "ffmpeg"
|
||||
AB_AV1_BIN = "ab-av1"
|
||||
|
||||
# Global state
|
||||
_shutdown_requested = False
|
||||
_lock_files = {}
|
||||
|
||||
def signal_handler(signum, frame):
|
||||
global _shutdown_requested
|
||||
_shutdown_requested = True
|
||||
print("\n\n[WARNING] Shutdown requested. Finishing current tasks...")
|
||||
|
||||
if platform.system() != "Windows":
|
||||
signal.signal(signal.SIGINT, signal_handler)
|
||||
signal.signal(signal.SIGTERM, signal_handler)
|
||||
|
||||
def run_command_streaming(cmd, description=""):
|
||||
"""Run command and stream output in real-time"""
|
||||
print(f" ▶ Running: {description}")
|
||||
|
||||
process = subprocess.Popen(
|
||||
cmd,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.STDOUT,
|
||||
universal_newlines=True,
|
||||
bufsize=1
|
||||
)
|
||||
|
||||
output_lines = []
|
||||
if process.stdout:
|
||||
for line in process.stdout:
|
||||
if _shutdown_requested:
|
||||
process.terminate()
|
||||
break
|
||||
line = line.rstrip()
|
||||
print(f" {line}")
|
||||
output_lines.append(line)
|
||||
|
||||
process.wait()
|
||||
return process.returncode, "\n".join(output_lines)
|
||||
|
||||
|
||||
def test_av1_sample(filepath, output_path):
|
||||
"""Test AV1 encoding on a sample segment"""
|
||||
print(f" 🧪 Testing AV1 with {SAMPLE_DURATION}s sample...")
|
||||
|
||||
cmd = [
|
||||
"ffmpeg",
|
||||
"-hide_banner",
|
||||
"-loglevel", "warning",
|
||||
"-ss", str(SAMPLE_START),
|
||||
"-i", str(filepath),
|
||||
"-t", str(SAMPLE_DURATION),
|
||||
"-c:v", "libsvtav1",
|
||||
"-crf", str(AV1_QUALITY),
|
||||
"-preset", "6",
|
||||
"-c:a", "copy",
|
||||
"-c:s", "copy",
|
||||
"-y",
|
||||
str(output_path)
|
||||
]
|
||||
|
||||
returncode, output = run_command_streaming(cmd, f"AV1 sample test")
|
||||
return returncode == 0 and output_path.exists()
|
||||
|
||||
|
||||
def encode_av1_full(filepath, output_path):
|
||||
"""Full AV1 encode using ab-av1"""
|
||||
print(f" 🎬 Full AV1 encode...")
|
||||
|
||||
cmd = [
|
||||
"ab-av1", "encode",
|
||||
"-i", str(filepath),
|
||||
"-o", str(output_path),
|
||||
"--crf", str(AV1_QUALITY),
|
||||
"--preset", "6",
|
||||
"--acodec", "copy"
|
||||
]
|
||||
|
||||
return run_command_streaming(cmd, f"AV1 full encode")[0] == 0
|
||||
|
||||
|
||||
def encode_hevc_full(filepath, output_path):
|
||||
"""Full HEVC encode using ffmpeg"""
|
||||
print(f" 🎬 Full HEVC encode...")
|
||||
|
||||
cmd = [
|
||||
"ffmpeg",
|
||||
"-hide_banner",
|
||||
"-loglevel", "warning",
|
||||
"-i", str(filepath),
|
||||
"-c:v", "libx265",
|
||||
"-crf", str(HEVC_QUALITY),
|
||||
"-preset", "medium",
|
||||
"-c:a", "copy",
|
||||
"-c:s", "copy",
|
||||
"-y",
|
||||
str(output_path)
|
||||
]
|
||||
|
||||
return run_command_streaming(cmd, f"HEVC full encode")[0] == 0
|
||||
|
||||
|
||||
def process_file(filepath, log_dir, log_category, lock_dir):
|
||||
"""Process a single video file with AV1→HEVC fallback"""
|
||||
filepath = Path(filepath)
|
||||
filename = filepath.name
|
||||
|
||||
# Check lock (using shared logic now)
|
||||
lock_file = common.acquire_lock(lock_dir, filepath, None) # Legacy encoder, no category root
|
||||
if not lock_file:
|
||||
print(f" 🔒 Skipping (locked): {filename}")
|
||||
return True
|
||||
|
||||
try:
|
||||
print(f"\n{'='*60}")
|
||||
print(f"📁 Processing: {filename}")
|
||||
print(f"{'='*60}")
|
||||
|
||||
# Get initial metadata
|
||||
metadata_before = common.get_video_info(filepath)
|
||||
if not metadata_before:
|
||||
print(f"❌ Could not read metadata, skipping")
|
||||
return False
|
||||
|
||||
print(f" 📊 Original:")
|
||||
print(f" Codec: {metadata_before['codec']}")
|
||||
print(f" Size: {metadata_before['size'] / (1024**3):.2f} GB")
|
||||
print(f" Bitrate: {metadata_before['bitrate']} kbps")
|
||||
print(f" Duration: {metadata_before['duration'] / 60:.1f} min")
|
||||
|
||||
# Skip if already AV1 or HEVC
|
||||
if metadata_before['codec'] in ['av1', 'hevc']:
|
||||
print(f" ℹ️ Already optimized ({metadata_before['codec']}), skipping")
|
||||
# We don't log skips to main log to avoid clutter, but could if needed
|
||||
return True
|
||||
|
||||
input_size = metadata_before['size']
|
||||
|
||||
# --- PHASE 1: AV1 SAMPLE TEST ---
|
||||
sample_output = TEMP_DIR / f"{filepath.stem}.sample.mkv"
|
||||
use_av1 = True
|
||||
|
||||
try:
|
||||
av1_test_passed = test_av1_sample(filepath, sample_output)
|
||||
|
||||
if av1_test_passed:
|
||||
sample_size = sample_output.stat().st_size
|
||||
sample_size_mb = sample_size / (1024 * 1024)
|
||||
|
||||
print(f" 📏 Sample size: {sample_size_mb:.1f} MB")
|
||||
|
||||
# Extrapolate to full file size
|
||||
duration_ratio = metadata_before['duration'] / SAMPLE_DURATION
|
||||
estimated_full_size = sample_size * duration_ratio
|
||||
estimated_full_gb = estimated_full_size / (1024**3)
|
||||
input_gb = input_size / (1024**3)
|
||||
|
||||
print(f" 📈 Estimated full size: {estimated_full_gb:.2f} GB")
|
||||
print(f" 📉 Original size: {input_gb:.2f} GB")
|
||||
|
||||
if sample_size_mb > MAX_SAMPLE_SIZE_MB:
|
||||
print(f" ❌ AV1 REJECTED: Sample too large ({sample_size_mb:.1f} MB > {MAX_SAMPLE_SIZE_MB} MB)")
|
||||
use_av1 = False
|
||||
elif estimated_full_size >= input_size:
|
||||
print(f" ❌ AV1 REJECTED: Estimated size ({estimated_full_gb:.2f} GB) >= original ({input_gb:.2f} GB)")
|
||||
use_av1 = False
|
||||
else:
|
||||
estimated_savings = (1 - estimated_full_size / input_size) * 100
|
||||
print(f" ✅ AV1 PASS: Estimated savings {estimated_savings:.1f}%")
|
||||
else:
|
||||
print(f" ❌ AV1 sample test failed")
|
||||
use_av1 = False
|
||||
|
||||
if sample_output.exists():
|
||||
sample_output.unlink()
|
||||
|
||||
except Exception as e:
|
||||
print(f" ❌ AV1 test error: {e}")
|
||||
use_av1 = False
|
||||
|
||||
# --- PHASE 2: ENCODE ---
|
||||
temp_output = TEMP_DIR / f"{filepath.stem}.temp.mkv"
|
||||
final_status = "rejected"
|
||||
final_codec = None
|
||||
final_size = input_size
|
||||
final_savings = 0.0
|
||||
|
||||
if use_av1:
|
||||
# Try AV1 full encode
|
||||
if encode_av1_full(filepath, temp_output):
|
||||
metadata_after = common.get_video_info(temp_output)
|
||||
if metadata_after:
|
||||
final_size = metadata_after['size']
|
||||
final_savings = (1 - final_size / input_size) * 100
|
||||
|
||||
if final_size < input_size:
|
||||
final_status = "success"
|
||||
final_codec = "av1"
|
||||
print(f" ✅ AV1 SUCCESS: Saved {final_savings:.1f}%")
|
||||
else:
|
||||
print(f" ❌ AV1 FAILED: Final size >= original")
|
||||
if temp_output.exists(): temp_output.unlink()
|
||||
|
||||
# Fall back to HEVC
|
||||
print(f" 🔄 Trying HEVC fallback...")
|
||||
if encode_hevc_full(filepath, temp_output):
|
||||
metadata_after = common.get_video_info(temp_output)
|
||||
if metadata_after:
|
||||
final_size = metadata_after['size']
|
||||
final_savings = (1 - final_size / input_size) * 100
|
||||
if final_size < input_size:
|
||||
final_status = "success"
|
||||
final_codec = "hevc"
|
||||
print(f" ✅ HEVC SUCCESS: Saved {final_savings:.1f}%")
|
||||
else:
|
||||
print(f" ❌ HEVC FAILED: Also larger than original")
|
||||
if temp_output.exists(): temp_output.unlink()
|
||||
else:
|
||||
print(f" ❌ AV1 encode failed, trying HEVC...")
|
||||
if encode_hevc_full(filepath, temp_output):
|
||||
metadata_after = common.get_video_info(temp_output)
|
||||
if metadata_after:
|
||||
final_size = metadata_after['size']
|
||||
final_savings = (1 - final_size / input_size) * 100
|
||||
if final_size < input_size:
|
||||
final_status = "success"
|
||||
final_codec = "hevc"
|
||||
print(f" ✅ HEVC SUCCESS: Saved {final_savings:.1f}%")
|
||||
else:
|
||||
print(f" ❌ HEVC FAILED: Larger than original")
|
||||
if temp_output.exists(): temp_output.unlink()
|
||||
else:
|
||||
# AV1 test failed, try HEVC directly
|
||||
print(f" 🔄 Trying HEVC directly...")
|
||||
if encode_hevc_full(filepath, temp_output):
|
||||
metadata_after = common.get_video_info(temp_output)
|
||||
if metadata_after:
|
||||
final_size = metadata_after['size']
|
||||
final_savings = (1 - final_size / input_size) * 100
|
||||
if final_size < input_size:
|
||||
final_status = "success"
|
||||
final_codec = "hevc"
|
||||
print(f" ✅ HEVC SUCCESS: Saved {final_savings:.1f}%")
|
||||
else:
|
||||
print(f" ❌ HEVC FAILED: Larger than original")
|
||||
if temp_output.exists(): temp_output.unlink()
|
||||
|
||||
# --- PHASE 3: FINALIZE ---
|
||||
if final_status == "success":
|
||||
# Replace original file (Safe Upload)
|
||||
if filepath.suffix:
|
||||
backup_path = filepath.with_suffix(f"{filepath.suffix}.backup")
|
||||
else:
|
||||
backup_path = Path(str(filepath) + ".backup")
|
||||
|
||||
shutil.move(str(filepath), str(backup_path))
|
||||
shutil.move(str(temp_output), str(filepath))
|
||||
|
||||
# Verify Integrity
|
||||
if Path(filepath).stat().st_size == final_size:
|
||||
backup_path.unlink()
|
||||
metadata_after = common.get_video_info(filepath)
|
||||
|
||||
common.log_event(log_dir, f"{log_category}.jsonl", {
|
||||
"file": str(filepath),
|
||||
"status": "success",
|
||||
"codec": final_codec,
|
||||
"input_size": input_size,
|
||||
"output_size": final_size,
|
||||
"savings_percent": final_savings,
|
||||
"metadata_before": metadata_before,
|
||||
"metadata_after": metadata_after
|
||||
})
|
||||
|
||||
print(f"\n ✅ SUCCESS: Optimized with {final_codec.upper() if final_codec else 'unknown'}")
|
||||
print(f" Savings: {final_savings:.1f}% ({input_size / (1024**3):.2f} GB → {final_size / (1024**3):.2f} GB)")
|
||||
else:
|
||||
print(" ❌ Critical Error: Copied file size mismatch! Restoring backup.")
|
||||
shutil.move(str(backup_path), str(filepath))
|
||||
|
||||
else:
|
||||
common.log_event(log_dir, "rejected.jsonl", {
|
||||
"file": str(filepath),
|
||||
"status": "rejected",
|
||||
"reason": "both_codecs_larger_than_original",
|
||||
"input_size": input_size,
|
||||
"metadata": metadata_before
|
||||
})
|
||||
|
||||
print(f"\n ❌ REJECTED: Both AV1 and HEVC larger than original")
|
||||
print(f" Keeping original file ({input_size / (1024**3):.2f} GB)")
|
||||
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
print(f"❌ Error processing {filename}: {e}")
|
||||
return False
|
||||
|
||||
finally:
|
||||
# Release shared lock
|
||||
if lock_file and lock_file.exists():
|
||||
lock_file.unlink()
|
||||
|
||||
def scan_directory(directory, extensions=None):
|
||||
"""Scan directory for video files"""
|
||||
if extensions is None:
|
||||
extensions = {".mkv", ".mp4", ".mov", ".avi", ".ts"}
|
||||
|
||||
files = []
|
||||
for dirpath, dirnames, filenames in os.walk(directory):
|
||||
# Skip processed/system files
|
||||
dirnames[:] = [d for d in dirnames if not d.startswith("_") and d not in [".recycle", "@eaDir"]]
|
||||
|
||||
for filename in filenames:
|
||||
filepath = Path(dirpath) / filename
|
||||
if filepath.suffix.lower() in extensions:
|
||||
if "_av1" in filepath.stem.lower() or "_hevc" in filepath.stem.lower():
|
||||
continue
|
||||
files.append(filepath)
|
||||
|
||||
return sorted(files)
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="Smart Video Encoder - AV1 with HEVC Fallback")
|
||||
parser.add_argument("--tv-dir", default=common.DEFAULT_CONFIG["tv_dir"], help="TV directory")
|
||||
parser.add_argument("--content-dir", default=common.DEFAULT_CONFIG["content_dir"], help="Content directory")
|
||||
parser.add_argument("--jobs", type=int, default=MAX_JOBS, help="Parallel jobs")
|
||||
parser.add_argument("--tv-only", action="store_true", help="Process TV only")
|
||||
parser.add_argument("--content-only", action="store_true", help="Process content only")
|
||||
args = parser.parse_args()
|
||||
|
||||
print("="*60)
|
||||
print("🎬 Smart Video Encoder - AV1 with HEVC Fallback")
|
||||
print("="*60)
|
||||
|
||||
# Setup
|
||||
common.check_dependencies()
|
||||
lock_dir, log_dir = common.get_base_paths(args)
|
||||
TEMP_DIR.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
print(f"\n📁 Configuration:")
|
||||
print(f" TV Directory: {args.tv_dir}")
|
||||
print(f" Content Directory: {args.content_dir}")
|
||||
print(f" Parallel Jobs: {args.jobs}")
|
||||
print(f" Locks: {lock_dir}")
|
||||
print(f" Logs: {log_dir}")
|
||||
print()
|
||||
|
||||
tasks = []
|
||||
if not args.content_only:
|
||||
tv_dir = Path(args.tv_dir)
|
||||
if tv_dir.exists():
|
||||
tv_files = scan_directory(tv_dir)
|
||||
print(f"📺 TV Files: {len(tv_files)}")
|
||||
for f in tv_files:
|
||||
tasks.append((f, log_dir, "tv_shows", lock_dir))
|
||||
|
||||
if not args.tv_only:
|
||||
content_dir = Path(args.content_dir)
|
||||
if content_dir.exists():
|
||||
content_files = scan_directory(content_dir)
|
||||
print(f"📦 Content Files: {len(content_files)}")
|
||||
for f in content_files:
|
||||
tasks.append((f, log_dir, "content", lock_dir))
|
||||
|
||||
if not tasks:
|
||||
print("❌ No files to process")
|
||||
return
|
||||
|
||||
print(f"\n🚀 Processing {len(tasks)} files...")
|
||||
print("="*60)
|
||||
|
||||
success_count = 0
|
||||
fail_count = 0
|
||||
|
||||
with ThreadPoolExecutor(max_workers=args.jobs) as executor:
|
||||
futures = {}
|
||||
for item in tasks:
|
||||
# item = (filepath, log_dir, log_category, lock_dir)
|
||||
future = executor.submit(process_file, *item)
|
||||
futures[future] = item[0]
|
||||
|
||||
for future in as_completed(futures):
|
||||
if _shutdown_requested:
|
||||
break
|
||||
|
||||
filepath = futures[future]
|
||||
try:
|
||||
result = future.result()
|
||||
if result:
|
||||
success_count += 1
|
||||
else:
|
||||
fail_count += 1
|
||||
except Exception as e:
|
||||
print(f"❌ Error processing {filepath}: {e}")
|
||||
fail_count += 1
|
||||
|
||||
print("\n" + "="*60)
|
||||
print("📊 Summary:")
|
||||
print(f" Processed: {success_count + fail_count}")
|
||||
print(f" ✅ Success: {success_count}")
|
||||
print(f" ❌ Failed: {fail_count}")
|
||||
print("="*60)
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
494
src/smart_gpu_encoder.py
Normal file
494
src/smart_gpu_encoder.py
Normal file
@@ -0,0 +1,494 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Smart GPU Encoder (Windows/AMD/NVIDIA Optimized)
|
||||
------------------------------------------------
|
||||
Refactored to use vmaf_common.py
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
import subprocess
|
||||
import json
|
||||
import shutil
|
||||
import time
|
||||
import argparse
|
||||
import signal
|
||||
import platform
|
||||
import re
|
||||
import threading
|
||||
from pathlib import Path
|
||||
from datetime import datetime
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
|
||||
# --- Import Common Module ---
|
||||
import vmaf_common as common
|
||||
|
||||
# --- Configuration ---
|
||||
TARGET_VMAF_MIN = common.DEFAULT_CONFIG["target_vmaf"]
|
||||
MIN_SAVINGS_PERCENT = common.DEFAULT_CONFIG["min_savings"]
|
||||
MAX_JOBS = common.DEFAULT_CONFIG["gpu_jobs"]
|
||||
|
||||
DEFAULT_AV1_QP = 32
|
||||
DEFAULT_HEVC_QP = 28
|
||||
|
||||
SAMPLE_DURATION = 60
|
||||
SAMPLE_START_TIME = 300
|
||||
|
||||
TEMP_DIR = None # Will be set in main
|
||||
|
||||
# Tools (Local override if needed, else used from common checks)
|
||||
FFMPEG_BIN = "ffmpeg"
|
||||
AB_AV1_BIN = "ab-av1"
|
||||
|
||||
# Global state
|
||||
shutdown_requested = False
|
||||
active_processes = set()
|
||||
upload_lock = threading.Lock()
|
||||
proc_lock = threading.Lock()
|
||||
debug_mode = False
|
||||
|
||||
def handle_sigint(signum, frame):
|
||||
global shutdown_requested
|
||||
print("\n\n[!] CRITICAL: Shutdown requested (Ctrl+C).")
|
||||
print(" Killing active encoder processes...")
|
||||
shutdown_requested = True
|
||||
|
||||
with proc_lock:
|
||||
for proc in list(active_processes):
|
||||
try:
|
||||
proc.terminate()
|
||||
time.sleep(0.1)
|
||||
if proc.poll() is None:
|
||||
proc.kill()
|
||||
except:
|
||||
pass
|
||||
print(" Cleanup complete. Exiting.")
|
||||
sys.exit(1)
|
||||
|
||||
signal.signal(signal.SIGINT, handle_sigint)
|
||||
|
||||
# --- Hardware Detection ---
|
||||
def detect_hardware_encoder():
|
||||
"""Detects available hardware encoders via ffmpeg"""
|
||||
# Use common module detection
|
||||
return common.detect_hardware_encoder()
|
||||
|
||||
def get_encoder_args(codec, encoder, qp):
|
||||
"""Returns correct ffmpeg args for specific HW vendor"""
|
||||
# Use common module args
|
||||
return common.get_encoder_args(codec, encoder, qp)
|
||||
|
||||
# --- Helpers ---
|
||||
def run_process(cmd, description="", status_callback=None):
|
||||
"""Run a process with real-time output and clean shutdown tracking"""
|
||||
if shutdown_requested: return False
|
||||
if status_callback: status_callback(description)
|
||||
|
||||
try:
|
||||
# Windows: Hide console window
|
||||
cflags = 0x08000000 if platform.system() == 'Windows' else 0
|
||||
|
||||
proc = subprocess.Popen(
|
||||
cmd,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.STDOUT,
|
||||
universal_newlines=True,
|
||||
bufsize=1,
|
||||
creationflags=cflags
|
||||
)
|
||||
|
||||
with proc_lock:
|
||||
active_processes.add(proc)
|
||||
|
||||
if proc.stdout:
|
||||
for line in proc.stdout:
|
||||
if shutdown_requested:
|
||||
proc.terminate()
|
||||
break
|
||||
line = line.strip()
|
||||
if line:
|
||||
if debug_mode: print(f" [Debug] {line}")
|
||||
|
||||
if status_callback and ("frame=" in line or "size=" in line or "time=" in line):
|
||||
status_callback(line)
|
||||
|
||||
proc.wait()
|
||||
|
||||
with proc_lock:
|
||||
if proc in active_processes:
|
||||
active_processes.remove(proc)
|
||||
|
||||
return proc.returncode == 0
|
||||
except Exception as e:
|
||||
if status_callback: status_callback(f"Error: {e}")
|
||||
return False
|
||||
|
||||
def run_vmaf_check(reference, distorted, status_callback=None):
|
||||
"""Run ab-av1 vmaf to get score"""
|
||||
# Use common dependency check to find binary if needed, but here just assume it's in path or bin
|
||||
ab_exe = "ab-av1"
|
||||
|
||||
# Check if bundled exists
|
||||
bundled = Path(__file__).parent / "bin" / "ab-av1.exe"
|
||||
if bundled.exists():
|
||||
ab_exe = str(bundled)
|
||||
|
||||
cmd = [ab_exe, "vmaf", "--reference", str(reference), "--distorted", str(distorted)]
|
||||
if status_callback: status_callback("Calculating VMAF...")
|
||||
|
||||
try:
|
||||
result = subprocess.run(cmd, capture_output=True, text=True, check=True)
|
||||
for line in result.stdout.splitlines():
|
||||
line = line.strip()
|
||||
match = re.search(r"VMAF\s+([0-9.]+)", line)
|
||||
if match: return float(match.group(1))
|
||||
try:
|
||||
val = float(line)
|
||||
if 0 <= val <= 100: return val
|
||||
except: pass
|
||||
return 0.0
|
||||
except Exception:
|
||||
return -1.0
|
||||
|
||||
# --- Core Logic ---
|
||||
def process_file(filepath, log_category, lock_dir, log_dir, encoders, category_root=None, worker_id=0, status_cb=None):
|
||||
"""
|
||||
Process a single file.
|
||||
status_cb: function(worker_id, filename, status_text, color)
|
||||
category_root: Root directory (tv_dir or content_dir) for relative path calculation
|
||||
"""
|
||||
av1_enc, hevc_enc, hw_type = encoders
|
||||
filepath = Path(filepath)
|
||||
filename = filepath.name
|
||||
|
||||
def update(msg, color="white"):
|
||||
if status_cb: status_cb(worker_id, filename, msg, color)
|
||||
else: print(f"[{worker_id}] {msg}")
|
||||
|
||||
if shutdown_requested: return
|
||||
|
||||
# 0. Check if already processed in a previous run
|
||||
if common.is_already_processed(log_dir, filepath):
|
||||
update("Already Processed (Skipping)", "dim")
|
||||
return
|
||||
|
||||
# 1. Lock Check (Shared Storage)
|
||||
lock_file = common.acquire_lock(lock_dir, filepath, category_root)
|
||||
if not lock_file:
|
||||
return # Locked or skipped
|
||||
|
||||
try:
|
||||
update("Analyzing...", "blue")
|
||||
|
||||
# 2. Analyze Source
|
||||
info = common.get_video_info(filepath)
|
||||
if not info:
|
||||
update("Metadata Error", "red")
|
||||
return
|
||||
|
||||
if info["codec"] == "av1":
|
||||
update("Already AV1 (Skipping)", "green")
|
||||
time.sleep(1)
|
||||
return
|
||||
|
||||
# 3. Create Samples
|
||||
sample_ref = TEMP_DIR / f"{filepath.stem}_{worker_id}_ref.mkv"
|
||||
sample_enc = TEMP_DIR / f"{filepath.stem}_{worker_id}_enc.mkv"
|
||||
|
||||
sample_start = SAMPLE_START_TIME
|
||||
if info["duration"] < (SAMPLE_START_TIME + SAMPLE_DURATION):
|
||||
sample_start = max(0, (info["duration"] / 2) - (SAMPLE_DURATION / 2))
|
||||
|
||||
update("Extracting Ref", "magenta")
|
||||
cmd_ref = [
|
||||
FFMPEG_BIN, "-y", "-hide_banner", "-loglevel", "error",
|
||||
"-ss", str(sample_start), "-t", str(SAMPLE_DURATION),
|
||||
"-i", str(filepath), "-c", "copy", "-map", "0:v:0",
|
||||
str(sample_ref)
|
||||
]
|
||||
if not run_process(cmd_ref):
|
||||
update("Extract Ref Failed", "red")
|
||||
return
|
||||
|
||||
# TEST 1: AV1
|
||||
vmaf_score = 0
|
||||
savings = 0
|
||||
|
||||
if av1_enc:
|
||||
update(f"Testing AV1 QP{DEFAULT_AV1_QP}", "yellow")
|
||||
enc_args = get_encoder_args("av1", av1_enc, DEFAULT_AV1_QP)
|
||||
cmd_enc = [
|
||||
FFMPEG_BIN, "-y", "-hide_banner", "-loglevel", "error",
|
||||
"-i", str(sample_ref), *enc_args, "-an", str(sample_enc)
|
||||
]
|
||||
|
||||
if run_process(cmd_enc):
|
||||
update("Calculating VMAF", "cyan")
|
||||
vmaf_score = run_vmaf_check(sample_ref, sample_enc)
|
||||
|
||||
ref_size = sample_ref.stat().st_size
|
||||
enc_size = sample_enc.stat().st_size
|
||||
savings = (1 - (enc_size / ref_size)) * 100 if ref_size > 0 else 0
|
||||
else:
|
||||
update("AV1 Test Failed", "red")
|
||||
|
||||
chosen_codec = None
|
||||
chosen_qp = 0
|
||||
|
||||
# Decision Logic
|
||||
if vmaf_score >= TARGET_VMAF_MIN and savings >= MIN_SAVINGS_PERCENT:
|
||||
chosen_codec = "av1"
|
||||
chosen_qp = DEFAULT_AV1_QP
|
||||
update(f"AV1 Good (VMAF {vmaf_score:.1f})", "green")
|
||||
|
||||
# Smart Optimization
|
||||
if vmaf_score > 97.0:
|
||||
update(f"Optimizing (High Quality {vmaf_score:.1f})", "yellow")
|
||||
new_qp = DEFAULT_AV1_QP + 4
|
||||
args_opt = get_encoder_args("av1", av1_enc, new_qp)
|
||||
cmd_opt = [FFMPEG_BIN, "-y", "-hide_banner", "-loglevel", "error", "-i", str(sample_ref), *args_opt, "-an", str(sample_enc)]
|
||||
|
||||
if run_process(cmd_opt):
|
||||
vmaf_opt = run_vmaf_check(sample_ref, sample_enc)
|
||||
size_opt = sample_enc.stat().st_size
|
||||
sav_opt = (1 - (size_opt / sample_ref.stat().st_size)) * 100
|
||||
|
||||
if vmaf_opt >= TARGET_VMAF_MIN and sav_opt > savings:
|
||||
update(f"Opt Accepted (+{sav_opt - savings:.1f}%)", "green")
|
||||
chosen_qp = new_qp
|
||||
vmaf_score = vmaf_opt
|
||||
savings = sav_opt
|
||||
else:
|
||||
update("Testing HEVC Fallback", "magenta")
|
||||
if info["codec"] != "hevc" and hevc_enc:
|
||||
hevc_args = get_encoder_args("hevc", hevc_enc, DEFAULT_HEVC_QP)
|
||||
cmd_hevc = [FFMPEG_BIN, "-y", "-hide_banner", "-loglevel", "error", "-i", str(sample_ref), *hevc_args, "-an", str(sample_enc)]
|
||||
run_process(cmd_hevc)
|
||||
|
||||
vmaf_score = run_vmaf_check(sample_ref, sample_enc)
|
||||
enc_size = sample_enc.stat().st_size
|
||||
savings = (1 - (enc_size / sample_ref.stat().st_size)) * 100
|
||||
|
||||
if vmaf_score >= TARGET_VMAF_MIN and savings >= MIN_SAVINGS_PERCENT:
|
||||
update(f"HEVC Accepted (VMAF {vmaf_score:.1f})", "green")
|
||||
chosen_codec = "hevc"
|
||||
chosen_qp = DEFAULT_HEVC_QP
|
||||
else:
|
||||
update("HEVC Rejected", "red")
|
||||
common.log_event(log_dir, "rejected.jsonl", {"file": str(filepath), "status": "rejected", "vmaf": vmaf_score})
|
||||
else:
|
||||
update("Skipping HEVC", "yellow")
|
||||
|
||||
# Cleanup Samples
|
||||
if sample_ref.exists(): sample_ref.unlink()
|
||||
if sample_enc.exists(): sample_enc.unlink()
|
||||
|
||||
# 4. Full Encode
|
||||
if chosen_codec:
|
||||
update(f"Encoding {chosen_codec.upper()} (QP {chosen_qp})", "green")
|
||||
output_file = TEMP_DIR / f"{filepath.stem}.{chosen_codec}.mkv"
|
||||
final_args = get_encoder_args(chosen_codec, av1_enc if chosen_codec=="av1" else hevc_enc, chosen_qp)
|
||||
|
||||
cmd_full = [
|
||||
FFMPEG_BIN, "-y", "-hide_banner", "-loglevel", "info", "-stats",
|
||||
"-i", str(filepath),
|
||||
*final_args,
|
||||
"-c:a", "copy", "-c:s", "copy", "-map", "0",
|
||||
str(output_file)
|
||||
]
|
||||
|
||||
def prog_cb(msg):
|
||||
if "frame=" in msg:
|
||||
try:
|
||||
if "time=" in msg:
|
||||
t_str = msg.split("time=")[1].split(" ")[0]
|
||||
h, m, s = map(float, t_str.split(':'))
|
||||
cur_sec = h*3600 + m*60 + s
|
||||
percent = (cur_sec / info["duration"]) * 100
|
||||
else:
|
||||
percent = 0.0
|
||||
|
||||
speed = "1x"
|
||||
if "speed=" in msg:
|
||||
speed = msg.split("speed=")[1].split("x")[0] + "x"
|
||||
|
||||
update(f"Encoding {chosen_codec} | {percent:.1f}% | {speed}", "green")
|
||||
except:
|
||||
pass
|
||||
|
||||
if run_process(cmd_full, f"Full Encode ({chosen_codec.upper()} QP {chosen_qp})", status_callback=prog_cb):
|
||||
final_info = common.get_video_info(output_file)
|
||||
if not final_info: final_info = {"size": output_file.stat().st_size}
|
||||
|
||||
final_size = final_info["size"]
|
||||
final_savings = (1 - (final_size / info["size"])) * 100
|
||||
saved_bytes = info["size"] - final_size
|
||||
|
||||
update(f"Uploading (Saved {final_savings:.1f}%)", "blue")
|
||||
|
||||
# UPLOAD (Serialized)
|
||||
with upload_lock:
|
||||
update(f"Uploading...", "blue")
|
||||
backup_path = filepath.with_suffix(f"{filepath.suffix}.original")
|
||||
try:
|
||||
shutil.move(str(filepath), str(backup_path))
|
||||
shutil.copy2(str(output_file), str(filepath))
|
||||
|
||||
# Verify integrity
|
||||
if Path(filepath).stat().st_size == final_size:
|
||||
output_file.unlink()
|
||||
backup_path.unlink()
|
||||
|
||||
# Refresh metadata for accuracy
|
||||
final_info_verified = common.get_video_info(filepath)
|
||||
|
||||
common.log_event(log_dir, f"{log_category}.jsonl", {
|
||||
"file": str(filepath),
|
||||
"status": "success",
|
||||
"codec": chosen_codec,
|
||||
"vmaf": vmaf_score,
|
||||
"savings": final_savings,
|
||||
"original_metadata": info,
|
||||
"encoded_metadata": final_info_verified or final_info
|
||||
})
|
||||
|
||||
# Mark as processed to prevent re-encoding in future runs
|
||||
common.mark_processed(log_dir, filepath, chosen_codec, vmaf_score, final_savings)
|
||||
|
||||
# Mark lock as completed (keep it for future runs)
|
||||
common.mark_lock_completed(lock_file)
|
||||
|
||||
update("Done", "green")
|
||||
if status_cb: status_cb(worker_id, filename, f"STATS:SAVED:{saved_bytes}", "green")
|
||||
else:
|
||||
update("Upload Failed (Size Mismatch)", "red")
|
||||
shutil.move(str(backup_path), str(filepath))
|
||||
except Exception as e:
|
||||
update(f"Move Error: {str(e)[:20]}", "red")
|
||||
if backup_path.exists(): shutil.move(str(backup_path), str(filepath))
|
||||
else:
|
||||
update("Encode Failed", "red")
|
||||
if output_file.exists(): output_file.unlink()
|
||||
|
||||
except Exception as e:
|
||||
update(f"Error: {str(e)[:30]}", "red")
|
||||
# On error, delete lock so file can be retried
|
||||
if lock_file and lock_file.exists():
|
||||
lock_file.unlink()
|
||||
finally:
|
||||
update("Idle", "dim")
|
||||
|
||||
def main():
|
||||
global debug_mode
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--tv-dir", default=common.DEFAULT_CONFIG["tv_dir"])
|
||||
parser.add_argument("--content-dir", default=common.DEFAULT_CONFIG["content_dir"])
|
||||
parser.add_argument("--jobs", type=int, default=MAX_JOBS)
|
||||
parser.add_argument("--debug", action="store_true")
|
||||
parser.add_argument("--skip-until", help="Skip all files alphabetically until this filename substring is found")
|
||||
parser.add_argument("--cpu-only", action="store_true", help="Force software encoding (CPU only)")
|
||||
parser.add_argument("--temp-dir", help="Override local temp directory")
|
||||
parser.add_argument("--av1-encoder", choices=["hw", "sw", "off"], default="hw", help="AV1 encoder: hw (hardware), sw (software), off (disable)")
|
||||
parser.add_argument("--hevc-encoder", choices=["hw", "sw", "off"], default="hw", help="HEVC encoder: hw (hardware), sw (software), off (disable)")
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.debug:
|
||||
debug_mode = True
|
||||
print("[Debug Mode Enabled]")
|
||||
|
||||
# 0. Check Dependencies
|
||||
common.check_dependencies()
|
||||
|
||||
# 1. Setup Directories
|
||||
lock_dir, log_dir = common.get_base_paths(args)
|
||||
|
||||
global TEMP_DIR
|
||||
TEMP_DIR = common.get_temp_dir(args)
|
||||
|
||||
# 2. Detect Hardware
|
||||
av1, hevc, hw = common.detect_hardware_encoder(args)
|
||||
|
||||
print("="*60)
|
||||
print(f" SMART ENCODER | Hardware: {hw.upper()} | Jobs: {args.jobs}")
|
||||
|
||||
if hw == "cpu":
|
||||
print(f" [!] Fallback to CPU Software Encoding (Slow)")
|
||||
if av1: print(f" AV1: {av1} (libsvtav1)")
|
||||
if hevc: print(f" HEVC: {hevc} (libx265)")
|
||||
else:
|
||||
print(f" AV1: {av1} | HEVC: {hevc}")
|
||||
|
||||
print(f" Locks: {lock_dir}")
|
||||
print("="*60)
|
||||
|
||||
# 3. Scan & Queue
|
||||
tasks = []
|
||||
|
||||
# Skip-until filtering
|
||||
skip_until = args.skip_until
|
||||
skipping = bool(skip_until)
|
||||
skipped_count = 0
|
||||
|
||||
tv_path = Path(args.tv_dir)
|
||||
if tv_path.exists():
|
||||
print(f"Scanning TV: {tv_path}")
|
||||
files = list(tv_path.rglob("*.mkv")) + list(tv_path.rglob("*.mp4"))
|
||||
files.sort() # Alphabetical order for consistency across platforms
|
||||
for f in files:
|
||||
if skipping:
|
||||
if skip_until.lower() in str(f).lower():
|
||||
skipping = False
|
||||
print(f" Found '{skip_until}' - resuming from here")
|
||||
else:
|
||||
skipped_count += 1
|
||||
continue
|
||||
tasks.append((f, "tv_shows"))
|
||||
|
||||
content_path = Path(args.content_dir)
|
||||
if content_path.exists():
|
||||
print(f"Scanning Content: {content_path}")
|
||||
files = list(content_path.rglob("*.mkv")) + list(content_path.rglob("*.mp4"))
|
||||
files.sort() # Alphabetical order for consistency across platforms
|
||||
for f in files:
|
||||
if skipping:
|
||||
if skip_until.lower() in str(f).lower():
|
||||
skipping = False
|
||||
print(f" Found '{skip_until}' - resuming from here")
|
||||
else:
|
||||
skipped_count += 1
|
||||
continue
|
||||
tasks.append((f, "content"))
|
||||
|
||||
if skipped_count > 0:
|
||||
print(f" Skipped {skipped_count} files (--skip-until)")
|
||||
|
||||
if not tasks:
|
||||
print("No files found.")
|
||||
return
|
||||
|
||||
# 4. Execute
|
||||
print(f"\n🚀 Processing {len(tasks)} files...")
|
||||
|
||||
# Build category root map
|
||||
category_roots = {
|
||||
"tv_shows": Path(args.tv_dir),
|
||||
"content": Path(args.content_dir)
|
||||
}
|
||||
|
||||
with ThreadPoolExecutor(max_workers=args.jobs) as executor:
|
||||
futures = {
|
||||
executor.submit(process_file, f, cat, lock_dir, log_dir, (av1, hevc, hw), category_roots.get(cat)): f
|
||||
for f, cat in tasks
|
||||
}
|
||||
|
||||
for future in as_completed(futures):
|
||||
if shutdown_requested:
|
||||
executor.shutdown(wait=False, cancel_futures=True)
|
||||
break
|
||||
try:
|
||||
future.result()
|
||||
except Exception as e:
|
||||
print(f"Worker Error: {e}")
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
427
src/smart_monitor.py
Normal file
427
src/smart_monitor.py
Normal file
@@ -0,0 +1,427 @@
|
||||
#!/usr/bin/env python3
|
||||
r"""
|
||||
Smart VMAF Monitor & Dashboard
|
||||
------------------------------
|
||||
The main interface for the VMAF Optimizer.
|
||||
Provides a TUI (Text User Interface) to visualize encoding progress.
|
||||
|
||||
Usage:
|
||||
python smart_monitor.py --tv-dir "Z:\tv" --content-dir "Z:\content" --jobs 4 [--monitor]
|
||||
"""
|
||||
|
||||
import os
|
||||
import time
|
||||
import argparse
|
||||
import sys
|
||||
import threading
|
||||
import queue
|
||||
import json
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
# Import the engine
|
||||
import smart_gpu_encoder as encoder
|
||||
import vmaf_common as common
|
||||
|
||||
# UI Library
|
||||
try:
|
||||
from rich.console import Console
|
||||
from rich.live import Live
|
||||
from rich.table import Table
|
||||
from rich.layout import Layout
|
||||
from rich.panel import Panel
|
||||
from rich.text import Text
|
||||
HAS_RICH = True
|
||||
except ImportError:
|
||||
HAS_RICH = False
|
||||
print("Warning: 'rich' library not found. Running in basic mode.")
|
||||
|
||||
# Watchdog
|
||||
try:
|
||||
from watchdog.observers import Observer
|
||||
from watchdog.events import FileSystemEventHandler
|
||||
HAS_WATCHDOG = True
|
||||
except ImportError:
|
||||
HAS_WATCHDOG = False
|
||||
|
||||
# --- Caching ---
|
||||
CACHE_FILE = Path("library_cache.json")
|
||||
|
||||
def load_cache():
|
||||
if CACHE_FILE.exists():
|
||||
try:
|
||||
with open(CACHE_FILE, "r") as f:
|
||||
return set(json.load(f))
|
||||
except: pass
|
||||
return set()
|
||||
|
||||
def save_cache(files):
|
||||
try:
|
||||
with open(CACHE_FILE, "w") as f:
|
||||
# Convert paths to strings for JSON
|
||||
json.dump([str(f) for f in files], f)
|
||||
except: pass
|
||||
|
||||
def fast_scan(path):
|
||||
"""Recursive scan using scandir (faster than pathlib)"""
|
||||
files = []
|
||||
try:
|
||||
if not os.path.exists(path): return []
|
||||
for entry in os.scandir(path):
|
||||
if entry.is_dir():
|
||||
files.extend(fast_scan(entry.path))
|
||||
elif entry.is_file():
|
||||
if entry.name.lower().endswith(('.mkv', '.mp4')):
|
||||
if "_enc" not in entry.name and "_ref" not in entry.name:
|
||||
files.append(entry.path)
|
||||
except:
|
||||
pass
|
||||
# Sort alphabetically for consistent ordering across platforms
|
||||
return sorted(files)
|
||||
|
||||
# --- UI State ---
|
||||
class Dashboard:
|
||||
def __init__(self, num_workers, log_dir=None):
|
||||
self.num_workers = num_workers
|
||||
self.worker_status = {i: {"file": "Idle", "action": "Waiting", "progress": 0, "speed": "", "color": "dim"} for i in range(num_workers)}
|
||||
self.stats = {"processed": 0, "skipped": 0, "failed": 0, "rejected": 0, "savings_gb": 0.0}
|
||||
self.recent_completed = []
|
||||
self.lock = threading.Lock()
|
||||
self.log_dir = log_dir
|
||||
self.activity_log_file = None
|
||||
|
||||
# Open activity log file for streaming
|
||||
if log_dir:
|
||||
try:
|
||||
log_path = Path(log_dir) / "activity.log"
|
||||
self.activity_log_file = open(log_path, "a", encoding="utf-8")
|
||||
except Exception as e:
|
||||
print(f"[Warning] Could not open activity log: {e}")
|
||||
|
||||
def format_filename(self, filename):
|
||||
# Clean Sonarr format: {Series} - S{s}E{e} - {Title} {Quality}
|
||||
# Goal: Series S01E01 [Quality]
|
||||
try:
|
||||
# Match S01E01
|
||||
match = re.search(r"(.*?) - (S\d+E\d+) - .*? ((?:Bluray|WebDL|Remux|2160p|1080p|Proper).*)\.mkv", filename, re.IGNORECASE)
|
||||
if match:
|
||||
series = match.group(1)[:15] # Truncate series name
|
||||
s_e = match.group(2)
|
||||
quality = match.group(3).split()[0] # Just take first part of quality (Bluray-2160p)
|
||||
return f"{series} {s_e} [{quality}]"
|
||||
|
||||
# Fallback for simpler names
|
||||
if len(filename) > 35:
|
||||
return filename[:15] + "..." + filename[-15:]
|
||||
return filename
|
||||
except:
|
||||
return filename
|
||||
|
||||
def update_worker(self, worker_id, file, action, color="blue"):
|
||||
with self.lock:
|
||||
display_name = self.format_filename(file)
|
||||
|
||||
progress = 0
|
||||
speed = ""
|
||||
|
||||
# Parse rich status: "Encoding AV1 | 45.2% | 2.3x"
|
||||
if "|" in action:
|
||||
parts = action.split("|")
|
||||
action_text = parts[0].strip()
|
||||
for p in parts[1:]:
|
||||
p = p.strip()
|
||||
if "%" in p:
|
||||
try: progress = float(p.replace("%", ""))
|
||||
except: pass
|
||||
elif "x" in p or "MB/s" in p:
|
||||
speed = p
|
||||
action = action_text
|
||||
|
||||
self.worker_status[worker_id] = {
|
||||
"file": display_name,
|
||||
"action": action,
|
||||
"progress": progress,
|
||||
"speed": speed,
|
||||
"color": color
|
||||
}
|
||||
|
||||
def add_log(self, message):
|
||||
with self.lock:
|
||||
ts = time.strftime("%Y-%m-%d %H:%M:%S")
|
||||
ts_short = time.strftime("%H:%M:%S")
|
||||
log_entry = f"[{ts_short}] {message}"
|
||||
|
||||
self.recent_completed.insert(0, log_entry)
|
||||
if len(self.recent_completed) > 12:
|
||||
self.recent_completed.pop()
|
||||
|
||||
# Stream to activity log file
|
||||
if self.activity_log_file:
|
||||
try:
|
||||
self.activity_log_file.write(f"[{ts}] {message}\n")
|
||||
self.activity_log_file.flush()
|
||||
except:
|
||||
pass
|
||||
|
||||
def update_stats(self, key, val=1):
|
||||
with self.lock:
|
||||
if key == "savings_gb": self.stats[key] += val
|
||||
else: self.stats[key] += val
|
||||
|
||||
def get_renderable(self):
|
||||
if not HAS_RICH: return ""
|
||||
|
||||
layout = Layout()
|
||||
layout.split_column(
|
||||
Layout(name="header", size=3),
|
||||
Layout(name="workers", size=self.num_workers + 4),
|
||||
Layout(name="stats", size=3),
|
||||
Layout(name="logs", ratio=1)
|
||||
)
|
||||
|
||||
layout["header"].update(Panel(Text("Smart GPU Encoder (AMD AMF + VMAF)", justify="center", style="bold cyan")))
|
||||
|
||||
# Workers Table
|
||||
table = Table(box=None, expand=True, padding=(0, 1))
|
||||
table.add_column("ID", width=3, style="dim", no_wrap=True)
|
||||
table.add_column("File", ratio=6, no_wrap=True)
|
||||
table.add_column("Action", ratio=3, no_wrap=True)
|
||||
table.add_column("Progress", width=20, no_wrap=True)
|
||||
table.add_column("Speed", width=12, no_wrap=True)
|
||||
|
||||
for i in range(self.num_workers):
|
||||
ws = self.worker_status[i]
|
||||
|
||||
pct = ws["progress"]
|
||||
# Rich Bar
|
||||
if pct > 0:
|
||||
bar_len = 12
|
||||
filled = int(bar_len * (pct / 100))
|
||||
bar_str = "━" * filled + " " * (bar_len - filled)
|
||||
prog_render = Text(f"{bar_str} {pct:.1f}%", style="green")
|
||||
else:
|
||||
prog_render = Text("")
|
||||
|
||||
table.add_row(
|
||||
str(i+1),
|
||||
Text(ws["file"], style="white"),
|
||||
Text(ws["action"], style=ws["color"]),
|
||||
prog_render,
|
||||
Text(ws["speed"], style="yellow")
|
||||
)
|
||||
|
||||
layout["workers"].update(Panel(table, title="Active Workers"))
|
||||
|
||||
stat_str = f"Processed: [green]{self.stats['processed']}[/] | Skipped: [yellow]{self.stats['skipped']}[/] | Rejected: [magenta]{self.stats['rejected']}[/] | Failed: [red]{self.stats['failed']}[/] | Saved: [bold green]{self.stats['savings_gb']:.2f} GB[/]"
|
||||
layout["stats"].update(Panel(Text.from_markup(stat_str, justify="center")))
|
||||
|
||||
layout["logs"].update(Panel("\n".join(self.recent_completed), title="Activity Log"))
|
||||
|
||||
return layout
|
||||
|
||||
# --- Worker Bridge ---
|
||||
def worker_wrapper(worker_id, file_path, category, category_root, lock_dir, log_dir, encoders, dashboard):
|
||||
def status_callback(w_id, fname, msg, color):
|
||||
# Handle Stats Signal
|
||||
if msg.startswith("STATS:"):
|
||||
parts = msg.split(":")
|
||||
if parts[1] == "SAVED":
|
||||
try:
|
||||
bytes_saved = float(parts[2])
|
||||
dashboard.update_stats("savings_gb", bytes_saved / (1024**3))
|
||||
dashboard.update_stats("processed")
|
||||
except: pass
|
||||
return
|
||||
|
||||
# Update Live Table
|
||||
dashboard.update_worker(worker_id, fname, msg, color)
|
||||
|
||||
# Check for Completion Events to update History
|
||||
# Match keywords from smart_gpu_encoder.py
|
||||
if "Done" in msg:
|
||||
dashboard.add_log(f"[Success] {dashboard.format_filename(fname)}")
|
||||
elif "Skipped" in msg or "Skipping" in msg:
|
||||
dashboard.add_log(f"[Skip] {dashboard.format_filename(fname)}")
|
||||
dashboard.update_stats("skipped")
|
||||
elif "Rejected" in msg:
|
||||
dashboard.add_log(f"[Reject] {dashboard.format_filename(fname)}")
|
||||
dashboard.update_stats("rejected")
|
||||
# Rejected doesn't count as failed, just processed (or ignored stats)
|
||||
elif "Failed" in msg or "Error" in msg:
|
||||
dashboard.add_log(f"[Fail] {dashboard.format_filename(fname)}")
|
||||
dashboard.update_stats("failed")
|
||||
|
||||
try:
|
||||
encoder.process_file(file_path, category, lock_dir, log_dir, encoders, category_root, worker_id, status_callback)
|
||||
except Exception as e:
|
||||
dashboard.add_log(f"Error in worker {worker_id}: {str(e)[:30]}")
|
||||
dashboard.update_stats("failed")
|
||||
finally:
|
||||
dashboard.update_worker(worker_id, "Idle", "Waiting", "dim")
|
||||
|
||||
# --- Monitor ---
|
||||
if HAS_WATCHDOG:
|
||||
class WatcherHandler(FileSystemEventHandler):
|
||||
def __init__(self, queue):
|
||||
self.queue = queue
|
||||
def on_created(self, event):
|
||||
if not event.is_directory and event.src_path.lower().endswith(('.mkv', '.mp4')):
|
||||
self.queue.put(Path(event.src_path))
|
||||
def on_moved(self, event):
|
||||
if not event.is_directory and event.dest_path.lower().endswith(('.mkv', '.mp4')):
|
||||
self.queue.put(Path(event.dest_path))
|
||||
else:
|
||||
# Dummy class when watchdog not available
|
||||
class WatcherHandler:
|
||||
def __init__(self, queue):
|
||||
self.queue = queue
|
||||
|
||||
# --- Main ---
|
||||
def main():
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--tv-dir", default=common.DEFAULT_CONFIG["tv_dir"])
|
||||
parser.add_argument("--content-dir", default=common.DEFAULT_CONFIG["content_dir"])
|
||||
parser.add_argument("--jobs", type=int, default=4)
|
||||
parser.add_argument("--monitor", action="store_true")
|
||||
parser.add_argument("--cpu-only", action="store_true", help="Force software encoding")
|
||||
parser.add_argument("--temp-dir", help="Override local temp directory")
|
||||
parser.add_argument("--skip-until", help="Skip files until this substring is found in filename")
|
||||
parser.add_argument("--av1-encoder", choices=["hw", "sw", "off"], default="hw", help="AV1 encoder: hw (hardware), sw (software), off (disable)")
|
||||
parser.add_argument("--hevc-encoder", choices=["hw", "sw", "off"], default="hw", help="HEVC encoder: hw (hardware), sw (software), off (disable)")
|
||||
args = parser.parse_args()
|
||||
|
||||
# Setup
|
||||
lock_dir, log_dir = common.get_base_paths(args)
|
||||
encoder.TEMP_DIR = common.get_temp_dir(args)
|
||||
|
||||
# Detect HW
|
||||
encoders = common.detect_hardware_encoder(args)
|
||||
av1_enc, hevc_enc, hw_type = encoders
|
||||
|
||||
# UI
|
||||
dashboard = Dashboard(args.jobs, log_dir=log_dir)
|
||||
dashboard.add_log(f"Locks: {lock_dir}")
|
||||
dashboard.add_log(f"Logs: {log_dir}")
|
||||
dashboard.add_log(f"Hardware: {hw_type.upper()}")
|
||||
dashboard.add_log(f"AV1: {av1_enc or 'None'} | HEVC: {hevc_enc or 'None'}")
|
||||
|
||||
# Work Queue
|
||||
work_queue = queue.Queue()
|
||||
|
||||
# Background Scanner
|
||||
def background_scanner():
|
||||
time.sleep(2) # Let UI start
|
||||
dashboard.add_log("Starting background scan...")
|
||||
|
||||
# Skip-until logic
|
||||
skip_until = args.skip_until
|
||||
skipping = bool(skip_until) # True until we find the match
|
||||
skipped_count = 0
|
||||
|
||||
if skip_until:
|
||||
dashboard.add_log(f"Skip mode: Looking for '{skip_until}'")
|
||||
|
||||
def should_skip(filepath):
|
||||
nonlocal skipping, skipped_count
|
||||
if not skipping:
|
||||
return False
|
||||
# Check if skip_until substring is in the filepath
|
||||
if skip_until.lower() in str(filepath).lower():
|
||||
skipping = False # Found it, stop skipping
|
||||
dashboard.add_log(f"Match found: {Path(filepath).name}")
|
||||
dashboard.add_log(f"Resuming processing from here")
|
||||
return False
|
||||
skipped_count += 1
|
||||
return True
|
||||
|
||||
# Load Cache first
|
||||
cached_files = load_cache()
|
||||
if cached_files:
|
||||
dashboard.add_log(f"Loaded {len(cached_files)} files from cache.")
|
||||
for f in cached_files:
|
||||
if should_skip(f):
|
||||
continue
|
||||
p = Path(f)
|
||||
cat = "tv_shows" if str(args.tv_dir) in str(p) else "content"
|
||||
work_queue.put((p, cat))
|
||||
|
||||
# Real Scan
|
||||
all_files = []
|
||||
for d, cat in [(args.tv_dir, "tv_shows"), (args.content_dir, "content")]:
|
||||
found = fast_scan(d)
|
||||
for f in found:
|
||||
all_files.append(f)
|
||||
# Only add if NOT in cache
|
||||
if str(f) not in cached_files:
|
||||
if should_skip(f):
|
||||
continue
|
||||
work_queue.put((Path(f), cat))
|
||||
|
||||
if skipped_count > 0:
|
||||
dashboard.add_log(f"Skipped {skipped_count} files (--skip-until)")
|
||||
|
||||
dashboard.add_log(f"Scan complete. Total: {len(all_files)}")
|
||||
save_cache(all_files)
|
||||
|
||||
# Start Scanner Thread
|
||||
scan_thread = threading.Thread(target=background_scanner, daemon=True)
|
||||
scan_thread.start()
|
||||
|
||||
# Thread Pool for Workers
|
||||
threads = []
|
||||
|
||||
def worker_loop(w_id):
|
||||
# Map category to root path
|
||||
category_roots = {
|
||||
"tv_shows": Path(args.tv_dir),
|
||||
"content": Path(args.content_dir)
|
||||
}
|
||||
|
||||
while not encoder.shutdown_requested:
|
||||
try:
|
||||
item = work_queue.get(timeout=1)
|
||||
file_path, category = item
|
||||
category_root = category_roots.get(category)
|
||||
worker_wrapper(w_id, file_path, category, category_root, lock_dir, log_dir, encoders, dashboard)
|
||||
work_queue.task_done()
|
||||
except queue.Empty:
|
||||
# If batch mode and scan is done and queue is empty, exit
|
||||
if not args.monitor and not scan_thread.is_alive() and work_queue.empty():
|
||||
time.sleep(2) # Grace period
|
||||
if work_queue.empty():
|
||||
return # Exit thread
|
||||
continue
|
||||
except Exception as e:
|
||||
dashboard.add_log(f"Worker {w_id} crashed: {e}")
|
||||
|
||||
for i in range(args.jobs):
|
||||
t = threading.Thread(target=worker_loop, args=(i,), daemon=True)
|
||||
t.start()
|
||||
threads.append(t)
|
||||
|
||||
# UI Loop
|
||||
try:
|
||||
if HAS_RICH:
|
||||
with Live(dashboard.get_renderable(), refresh_per_second=4) as live:
|
||||
while not encoder.shutdown_requested:
|
||||
live.update(dashboard.get_renderable())
|
||||
time.sleep(0.25)
|
||||
|
||||
# Exit condition
|
||||
if not args.monitor and not scan_thread.is_alive() and work_queue.unfinished_tasks == 0:
|
||||
# Check if threads are actually dead
|
||||
if all(not t.is_alive() for t in threads):
|
||||
break
|
||||
else:
|
||||
while not encoder.shutdown_requested:
|
||||
time.sleep(1)
|
||||
if not args.monitor and not scan_thread.is_alive() and work_queue.empty(): break
|
||||
|
||||
except KeyboardInterrupt:
|
||||
encoder.shutdown_requested = True
|
||||
print("\nStopping...")
|
||||
|
||||
print("\nDone.")
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
47
src/test_monitor_logic.py
Normal file
47
src/test_monitor_logic.py
Normal file
@@ -0,0 +1,47 @@
|
||||
from smart_monitor import Dashboard
|
||||
import time
|
||||
|
||||
def test_tui_parsing():
|
||||
print("Testing Dashboard Logic...")
|
||||
db = Dashboard(1)
|
||||
|
||||
# Test 1: Encoding Progress
|
||||
msg = "Encoding av1 | 45.2% | 2.3x"
|
||||
db.update_worker(0, "Test File 1.mkv", msg)
|
||||
status = db.worker_status[0]
|
||||
print(f"Test 1 (Parsing): Progress={status['progress']} (Expected 45.2), Speed={status['speed']} (Expected 2.3x)")
|
||||
assert status['progress'] == 45.2
|
||||
assert status['speed'] == "2.3x"
|
||||
|
||||
# Test 2: Stats - Skipped
|
||||
print("\nTest 2: Stats - Skipped")
|
||||
# Simulate worker wrapper logic
|
||||
msg = "Already AV1 (Skipping)"
|
||||
if "Skipping" in msg:
|
||||
db.update_stats("skipped")
|
||||
|
||||
print(f"Skipped Count: {db.stats['skipped']} (Expected 1)")
|
||||
assert db.stats['skipped'] == 1
|
||||
|
||||
# Test 3: Stats - Rejected
|
||||
print("\nTest 3: Stats - Rejected")
|
||||
msg = "HEVC Rejected"
|
||||
if "Rejected" in msg:
|
||||
db.update_stats("rejected")
|
||||
|
||||
print(f"Rejected Count: {db.stats['rejected']} (Expected 1)")
|
||||
assert db.stats['rejected'] == 1
|
||||
|
||||
# Test 4: Stats - Failed
|
||||
print("\nTest 4: Stats - Failed")
|
||||
msg = "Error: FFMPEG failed"
|
||||
if "Error" in msg:
|
||||
db.update_stats("failed")
|
||||
|
||||
print(f"Failed Count: {db.stats['failed']} (Expected 1)")
|
||||
assert db.stats['failed'] == 1
|
||||
|
||||
print("\nAll Tests Passed!")
|
||||
|
||||
if __name__ == "__main__":
|
||||
test_tui_parsing()
|
||||
481
src/vmaf_common.py
Normal file
481
src/vmaf_common.py
Normal file
@@ -0,0 +1,481 @@
|
||||
import os
|
||||
import sys
|
||||
import shutil
|
||||
import json
|
||||
import hashlib
|
||||
import platform
|
||||
import subprocess
|
||||
import time
|
||||
from pathlib import Path
|
||||
from datetime import datetime
|
||||
|
||||
# --- Defaults ---
|
||||
DEFAULT_CONFIG = {
|
||||
"tv_dir": r"Z:\tv",
|
||||
"content_dir": r"Z:\content",
|
||||
"target_vmaf": 93.0,
|
||||
"min_savings": 12.0,
|
||||
"gpu_jobs": 2,
|
||||
"cpu_jobs": 1,
|
||||
"av1_crf": 34,
|
||||
"hevc_crf": 28,
|
||||
"temp_dir_windows": r"C:\Users\bnair\Videos\encodes",
|
||||
"temp_dir_macos": "/Users/bnair/Documents/encodes",
|
||||
"temp_dir_linux": "~/Videos/encodes"
|
||||
}
|
||||
|
||||
# --- Paths ---
|
||||
def get_base_paths(args=None):
|
||||
r"""
|
||||
Determine root paths for locks and logs.
|
||||
Priority:
|
||||
1. Shared Network Drive (parent of tv_dir) -> Z:\.vmaf_locks
|
||||
2. Local Fallback (current dir / logs)
|
||||
"""
|
||||
tv_dir = Path(args.tv_dir) if args and hasattr(args, 'tv_dir') else Path(DEFAULT_CONFIG["tv_dir"])
|
||||
|
||||
# Logic: Locks MUST be at the common root of content to be shared.
|
||||
# We attempt to find the common parent if both are supplied, otherwise default to tv_dir parent.
|
||||
|
||||
potential_roots = []
|
||||
if args:
|
||||
if hasattr(args, 'tv_dir') and args.tv_dir: potential_roots.append(Path(args.tv_dir))
|
||||
if hasattr(args, 'content_dir') and args.content_dir: potential_roots.append(Path(args.content_dir))
|
||||
|
||||
if not potential_roots:
|
||||
potential_roots.append(Path(DEFAULT_CONFIG["tv_dir"]))
|
||||
|
||||
# Find common path
|
||||
try:
|
||||
if len(potential_roots) > 1:
|
||||
# If they are on different drives, common_path might fail or return empty on Windows
|
||||
# In that case, we fall back to the first one (TV dir parent)
|
||||
common_root = os.path.commonpath(potential_roots)
|
||||
shared_root = Path(common_root)
|
||||
# If common root is the root drive itself (e.g. Z:\), that's fine.
|
||||
# If it's a subdir (Z:\Media), also fine.
|
||||
# But if it's too deep (e.g. just Z:\), we want to make sure we don't dump .vmaf_locks in root unless intended.
|
||||
# Typically, we want the parent of the library folders.
|
||||
# If os.path.commonpath returns Z:\Media, and dirs are Z:\Media\TV and Z:\Media\Content
|
||||
# Then locks go to Z:\Media\.vmaf_locks. Perfect.
|
||||
else:
|
||||
shared_root = potential_roots[0].parent
|
||||
except:
|
||||
# Fallback for different drives
|
||||
shared_root = potential_roots[0].parent
|
||||
|
||||
# Defaults
|
||||
lock_dir = Path("locks").resolve()
|
||||
log_dir = Path("logs").resolve()
|
||||
|
||||
# Network Logic
|
||||
if shared_root.exists():
|
||||
network_lock = shared_root / ".vmaf_locks"
|
||||
# We prefer local logs to avoid network I/O issues during logging,
|
||||
# but locks MUST be shared.
|
||||
try:
|
||||
network_lock.mkdir(parents=True, exist_ok=True)
|
||||
lock_dir = network_lock
|
||||
except Exception as e:
|
||||
print(f"[Warning] Could not create network locks at {network_lock}: {e}")
|
||||
print(f" Falling back to local locks: {lock_dir}")
|
||||
|
||||
# Ensure existence
|
||||
lock_dir.mkdir(parents=True, exist_ok=True)
|
||||
log_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
return lock_dir, log_dir
|
||||
|
||||
def get_temp_dir(args=None):
|
||||
# Check args override
|
||||
if args and hasattr(args, 'temp_dir') and args.temp_dir:
|
||||
path = Path(args.temp_dir)
|
||||
elif platform.system() == "Windows":
|
||||
path = Path(DEFAULT_CONFIG["temp_dir_windows"])
|
||||
elif platform.system() == "Darwin": # macOS
|
||||
path = Path(DEFAULT_CONFIG["temp_dir_macos"])
|
||||
else: # Linux
|
||||
path = Path(DEFAULT_CONFIG["temp_dir_linux"]).expanduser()
|
||||
|
||||
path.mkdir(parents=True, exist_ok=True)
|
||||
return path
|
||||
|
||||
# --- Logging ---
|
||||
def log_event(log_dir, filename, data):
|
||||
log_path = Path(log_dir) / filename
|
||||
data["timestamp"] = datetime.now().isoformat()
|
||||
try:
|
||||
with open(log_path, "a", encoding="utf-8") as f:
|
||||
f.write(json.dumps(data) + "\n")
|
||||
except Exception as e:
|
||||
print(f"[ERROR] Failed to write log: {e}")
|
||||
|
||||
# --- Global Processed Log ---
|
||||
# Prevents re-encoding files that were already processed in previous runs
|
||||
_processed_cache = None
|
||||
|
||||
def load_processed_log(log_dir):
|
||||
"""Load set of already-processed file paths from processed.jsonl"""
|
||||
global _processed_cache
|
||||
if _processed_cache is not None:
|
||||
return _processed_cache
|
||||
|
||||
_processed_cache = set()
|
||||
log_path = Path(log_dir) / "processed.jsonl"
|
||||
|
||||
if log_path.exists():
|
||||
try:
|
||||
with open(log_path, "r", encoding="utf-8") as f:
|
||||
for line in f:
|
||||
try:
|
||||
entry = json.loads(line.strip())
|
||||
if "file" in entry:
|
||||
_processed_cache.add(entry["file"])
|
||||
except:
|
||||
continue
|
||||
except Exception as e:
|
||||
print(f"[Warning] Could not load processed log: {e}")
|
||||
|
||||
return _processed_cache
|
||||
|
||||
def is_already_processed(log_dir, filepath):
|
||||
"""Check if file was already processed in a previous run"""
|
||||
processed = load_processed_log(log_dir)
|
||||
return str(filepath) in processed
|
||||
|
||||
def mark_processed(log_dir, filepath, codec, vmaf, savings):
|
||||
"""Mark a file as processed (prevents re-encoding in future runs)"""
|
||||
global _processed_cache
|
||||
log_path = Path(log_dir) / "processed.jsonl"
|
||||
|
||||
entry = {
|
||||
"file": str(filepath),
|
||||
"codec": codec,
|
||||
"vmaf": vmaf,
|
||||
"savings": savings,
|
||||
"timestamp": datetime.now().isoformat()
|
||||
}
|
||||
|
||||
try:
|
||||
with open(log_path, "a", encoding="utf-8") as f:
|
||||
f.write(json.dumps(entry) + "\n")
|
||||
|
||||
# Update cache
|
||||
if _processed_cache is not None:
|
||||
_processed_cache.add(str(filepath))
|
||||
except Exception as e:
|
||||
print(f"[Warning] Could not write to processed log: {e}")
|
||||
|
||||
# --- Dependencies ---
|
||||
def check_dependencies(required_tools=None):
|
||||
if required_tools is None:
|
||||
required_tools = ["ffmpeg", "ffprobe", "ab-av1"]
|
||||
|
||||
missing = []
|
||||
# Determine bin path relative to this file
|
||||
bin_path = Path(__file__).parent.parent / "bin"
|
||||
|
||||
for tool in required_tools:
|
||||
if tool == "ab-av1":
|
||||
# Check bundled first in bin/
|
||||
bundled = bin_path / "ab-av1.exe"
|
||||
if bundled.exists(): continue
|
||||
# Check PATH
|
||||
if shutil.which("ab-av1"): continue
|
||||
missing.append("ab-av1")
|
||||
else:
|
||||
if not shutil.which(tool):
|
||||
missing.append(tool)
|
||||
|
||||
if missing:
|
||||
print(f"[!] CRITICAL: Missing tools: {', '.join(missing)}")
|
||||
print(f" Checked bundled path: {bin_path}")
|
||||
sys.exit(1)
|
||||
|
||||
# --- Metadata ---
|
||||
def get_video_info(filepath):
|
||||
"""Get video info using ffprobe (Standardized)"""
|
||||
try:
|
||||
cmd = [
|
||||
"ffprobe", "-v", "quiet",
|
||||
"-print_format", "json",
|
||||
"-show_format", "-show_streams",
|
||||
str(filepath)
|
||||
]
|
||||
result = subprocess.run(cmd, capture_output=True, text=True, check=True)
|
||||
data = json.loads(result.stdout)
|
||||
|
||||
video = next((s for s in data["streams"] if s["codec_type"] == "video"), None)
|
||||
if not video: return None
|
||||
|
||||
size = int(data["format"].get("size", 0))
|
||||
duration = float(data["format"].get("duration", 0))
|
||||
|
||||
# Bitrate calc
|
||||
bitrate = int(data["format"].get("bitrate", 0))
|
||||
if bitrate == 0 and duration > 0:
|
||||
bitrate = int((size * 8) / duration)
|
||||
|
||||
return {
|
||||
"codec": video.get("codec_name"),
|
||||
"width": int(video.get("width", 0)),
|
||||
"height": int(video.get("height", 0)),
|
||||
"duration": duration,
|
||||
"size": size,
|
||||
"bitrate": bitrate,
|
||||
"fps": video.get("r_frame_rate", "0/0")
|
||||
}
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
# --- Locks ---
|
||||
def acquire_lock(lock_dir, filepath, category_root=None):
|
||||
"""
|
||||
Simple file-based lock with completion tracking.
|
||||
Uses relative path from category root for cross-platform compatibility.
|
||||
|
||||
Lock lifecycle:
|
||||
- Created with status="processing" when file starts
|
||||
- Updated to status="completed" when file finishes successfully
|
||||
- Deleted only if process is cancelled/interrupted
|
||||
|
||||
This way lock files act as "already processed" markers across machines.
|
||||
|
||||
Args:
|
||||
lock_dir: Directory to store lock files
|
||||
filepath: Full path to the video file
|
||||
category_root: Root directory (tv_dir or content_dir) to calculate relative path
|
||||
|
||||
Returns lock_path if acquired, None if failed/already completed.
|
||||
"""
|
||||
# Calculate relative path for consistent hashing across platforms
|
||||
if category_root:
|
||||
try:
|
||||
rel_path = Path(filepath).relative_to(category_root)
|
||||
# Use forward slashes for consistency across platforms
|
||||
hash_key = str(rel_path).replace(os.sep, '/')
|
||||
except ValueError:
|
||||
# File not under category_root, use filename
|
||||
hash_key = Path(filepath).name
|
||||
else:
|
||||
hash_key = Path(filepath).name
|
||||
|
||||
fhash = hashlib.md5(hash_key.encode()).hexdigest()
|
||||
lock_file = lock_dir / f"{fhash}.lock"
|
||||
|
||||
if lock_file.exists():
|
||||
try:
|
||||
# Read lock file to check status
|
||||
lock_data = json.loads(lock_file.read_text())
|
||||
|
||||
if lock_data.get("status") == "completed":
|
||||
# File already processed successfully - skip it
|
||||
return None
|
||||
|
||||
# Check if it's a stale "processing" lock (older than 24h)
|
||||
if lock_data.get("status") == "processing":
|
||||
if time.time() - lock_data.get("timestamp", 0) > 86400:
|
||||
# Stale lock - remove and retry
|
||||
lock_file.unlink()
|
||||
else:
|
||||
# Fresh processing lock - file is being worked on
|
||||
return None
|
||||
except:
|
||||
# Corrupted lock file - remove and retry
|
||||
lock_file.unlink()
|
||||
|
||||
try:
|
||||
# Create new processing lock
|
||||
import socket
|
||||
lock_info = {
|
||||
"file": str(filepath),
|
||||
"relative_path": hash_key,
|
||||
"host": socket.gethostname(),
|
||||
"timestamp": time.time(),
|
||||
"status": "processing"
|
||||
}
|
||||
lock_file.write_text(json.dumps(lock_info))
|
||||
return lock_file
|
||||
except Exception as e:
|
||||
print(f"[Warning] Failed to acquire lock for {filepath.name}: {e}")
|
||||
return None
|
||||
|
||||
def mark_lock_completed(lock_file):
|
||||
"""Mark a lock file as completed (keep it for future runs)"""
|
||||
if not lock_file or not lock_file.exists():
|
||||
return
|
||||
|
||||
try:
|
||||
lock_data = json.loads(lock_file.read_text())
|
||||
lock_data["status"] = "completed"
|
||||
lock_data["completed_at"] = time.time()
|
||||
lock_file.write_text(json.dumps(lock_data))
|
||||
except Exception as e:
|
||||
print(f"[Warning] Failed to mark lock as completed: {e}")
|
||||
|
||||
# --- Hardware Detection ---
|
||||
def detect_hardware_encoder(args=None):
|
||||
"""
|
||||
Detects available hardware encoders via ffmpeg (Cross-Platform).
|
||||
|
||||
Supports --av1-encoder and --hevc-encoder flags:
|
||||
hw = prefer hardware encoder (error if unavailable)
|
||||
sw = force software encoder
|
||||
off = disable this codec entirely
|
||||
"""
|
||||
|
||||
# Get user preferences from args
|
||||
av1_pref = getattr(args, 'av1_encoder', 'hw') if args else 'hw'
|
||||
hevc_pref = getattr(args, 'hevc_encoder', 'hw') if args else 'hw'
|
||||
cpu_only = getattr(args, 'cpu_only', False) if args else False
|
||||
|
||||
# If cpu_only, treat both as 'sw'
|
||||
if cpu_only:
|
||||
av1_pref = 'sw'
|
||||
hevc_pref = 'sw'
|
||||
|
||||
try:
|
||||
# Run ffmpeg -encoders
|
||||
res = subprocess.run(["ffmpeg", "-hide_banner", "-encoders"], capture_output=True, text=True)
|
||||
out = res.stdout
|
||||
|
||||
# Build available encoder maps
|
||||
hw_av1_encoders = []
|
||||
hw_hevc_encoders = []
|
||||
hw_type = "cpu"
|
||||
|
||||
# 1. AMD (AMF) - Windows (Preferred)
|
||||
if "av1_amf" in out: hw_av1_encoders.append(("av1_amf", "amf"))
|
||||
if "hevc_amf" in out: hw_hevc_encoders.append(("hevc_amf", "amf"))
|
||||
|
||||
# 2. NVIDIA (NVENC) - Windows/Linux
|
||||
if "av1_nvenc" in out: hw_av1_encoders.append(("av1_nvenc", "nvenc"))
|
||||
if "hevc_nvenc" in out: hw_hevc_encoders.append(("hevc_nvenc", "nvenc"))
|
||||
|
||||
# 3. AMD (VAAPI) - Linux
|
||||
if "av1_vaapi" in out: hw_av1_encoders.append(("av1_vaapi", "vaapi"))
|
||||
if "hevc_vaapi" in out: hw_hevc_encoders.append(("hevc_vaapi", "vaapi"))
|
||||
|
||||
# 4. Intel (QSV) - Windows/Linux
|
||||
if "av1_qsv" in out: hw_av1_encoders.append(("av1_qsv", "qsv"))
|
||||
if "hevc_qsv" in out: hw_hevc_encoders.append(("hevc_qsv", "qsv"))
|
||||
|
||||
# 5. Apple Silicon (VideoToolbox) - macOS
|
||||
if "av1_videotoolbox" in out: hw_av1_encoders.append(("av1_videotoolbox", "videotoolbox"))
|
||||
if "hevc_videotoolbox" in out: hw_hevc_encoders.append(("hevc_videotoolbox", "videotoolbox"))
|
||||
|
||||
# Software encoders
|
||||
has_libsvtav1 = "libsvtav1" in out
|
||||
has_libx265 = "libx265" in out
|
||||
|
||||
# Resolve AV1 encoder
|
||||
av1_enc = None
|
||||
if av1_pref == 'off':
|
||||
av1_enc = None
|
||||
elif av1_pref == 'sw':
|
||||
if has_libsvtav1:
|
||||
av1_enc = "libsvtav1"
|
||||
else:
|
||||
print("[Warning] libsvtav1 not available, AV1 disabled")
|
||||
elif av1_pref == 'hw':
|
||||
if hw_av1_encoders:
|
||||
av1_enc, hw_type = hw_av1_encoders[0] # First available (AMD priority)
|
||||
elif has_libsvtav1:
|
||||
av1_enc = "libsvtav1"
|
||||
print("[Info] No AV1 HW encoder, using libsvtav1 (CPU)")
|
||||
else:
|
||||
print("[Warning] No AV1 encoder available")
|
||||
|
||||
# Resolve HEVC encoder
|
||||
hevc_enc = None
|
||||
if hevc_pref == 'off':
|
||||
hevc_enc = None
|
||||
elif hevc_pref == 'sw':
|
||||
if has_libx265:
|
||||
hevc_enc = "libx265"
|
||||
else:
|
||||
print("[Warning] libx265 not available, HEVC disabled")
|
||||
elif hevc_pref == 'hw':
|
||||
if hw_hevc_encoders:
|
||||
hevc_enc, hevc_hw = hw_hevc_encoders[0]
|
||||
# Update hw_type if we didn't get one from AV1
|
||||
if hw_type == "cpu":
|
||||
hw_type = hevc_hw
|
||||
elif has_libx265:
|
||||
hevc_enc = "libx265"
|
||||
print("[Info] No HEVC HW encoder, using libx265 (CPU)")
|
||||
else:
|
||||
print("[Warning] No HEVC encoder available")
|
||||
|
||||
# Determine final hw_type label
|
||||
if av1_enc and "lib" not in av1_enc:
|
||||
pass # hw_type already set
|
||||
elif hevc_enc and "lib" not in hevc_enc:
|
||||
pass # hw_type already set from HEVC
|
||||
else:
|
||||
hw_type = "cpu"
|
||||
|
||||
return av1_enc, hevc_enc, hw_type
|
||||
|
||||
except Exception as e:
|
||||
print(f"[Warning] HW Detection failed: {e}")
|
||||
return None, None, "error"
|
||||
|
||||
def get_encoder_args(codec, encoder, qp):
|
||||
"""Returns correct ffmpeg args for specific HW vendor"""
|
||||
if not encoder: return []
|
||||
|
||||
# Software (CPU)
|
||||
if encoder == "libsvtav1":
|
||||
# CRF 0-63 (Lower is better)
|
||||
return ["-c:v", "libsvtav1", "-crf", str(qp), "-preset", "6", "-g", "240"]
|
||||
|
||||
if encoder == "libx265":
|
||||
return ["-c:v", "libx265", "-crf", str(qp), "-preset", "medium"]
|
||||
|
||||
# NVIDIA NVENC
|
||||
if "nvenc" in encoder:
|
||||
# p6 = better quality, spatial-aq for better perception
|
||||
return ["-c:v", encoder, "-rc", "constqp", "-qp", str(qp), "-preset", "p6", "-spatial-aq", "1"]
|
||||
|
||||
# AMD AMF
|
||||
if "amf" in encoder:
|
||||
return ["-c:v", encoder, "-usage", "transcoding", "-rc", "cqp", "-qp_i", str(qp), "-qp_p", str(qp), "-qp_b", str(qp), "-quality", "quality"]
|
||||
|
||||
# Intel QSV
|
||||
if "qsv" in encoder:
|
||||
return ["-c:v", encoder, "-global_quality", str(qp), "-look_ahead", "1"]
|
||||
|
||||
# Apple VideoToolbox
|
||||
if "videotoolbox" in encoder:
|
||||
# Map 0-51 QP to 100-0 Quality (approx)
|
||||
q = int(100 - (qp * 2))
|
||||
return ["-c:v", encoder, "-q:v", str(q)]
|
||||
|
||||
# Linux VAAPI
|
||||
if "vaapi" in encoder:
|
||||
# Uses -qp normally? or -global_quality? Depends on driver.
|
||||
# Often needs: -vf format=nv12,hwupload
|
||||
# Safe bet for vaapi is usually CQP via -qp or -global_quality
|
||||
return ["-c:v", encoder, "-qp", str(qp)]
|
||||
|
||||
return []
|
||||
|
||||
def scan_directory(directory, extensions=None):
|
||||
"""Scan directory for video files"""
|
||||
if extensions is None:
|
||||
extensions = {".mkv", ".mp4", ".mov", ".avi", ".ts"}
|
||||
|
||||
files = []
|
||||
for dirpath, dirnames, filenames in os.walk(directory):
|
||||
# Skip processed/system files
|
||||
dirnames[:] = [d for d in dirnames if not d.startswith("_") and d not in [".recycle", "@eaDir"]]
|
||||
|
||||
for filename in filenames:
|
||||
filepath = Path(dirpath) / filename
|
||||
if filepath.suffix.lower() in extensions:
|
||||
if "_av1" in filepath.stem.lower() or "_hevc" in filepath.stem.lower():
|
||||
continue
|
||||
files.append(filepath)
|
||||
|
||||
return sorted(files, key=lambda x: x.stat().st_size, reverse=True)
|
||||
42
test_run.py
Normal file
42
test_run.py
Normal file
@@ -0,0 +1,42 @@
|
||||
import os
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
# Setup
|
||||
TEST_DIR = Path("test_media")
|
||||
TEST_DIR.mkdir(exist_ok=True)
|
||||
TEST_FILE = TEST_DIR / "test_video.mkv"
|
||||
|
||||
print("--- SMART ENCODER TEST ---")
|
||||
|
||||
# 1. Create Dummy Video (10s, 1080p, noise to ensure it's not trivial)
|
||||
if not TEST_FILE.exists():
|
||||
print("Generating dummy video...")
|
||||
# Generate 10s video with noise
|
||||
cmd = [
|
||||
"ffmpeg", "-y", "-f", "lavfi", "-i", "testsrc=duration=10:size=1920x1080:rate=30",
|
||||
"-f", "lavfi", "-i", "sine=frequency=1000:duration=10",
|
||||
"-c:v", "libx264", "-pix_fmt", "yuv420p", "-b:v", "5M",
|
||||
"-c:a", "aac",
|
||||
str(TEST_FILE)
|
||||
]
|
||||
subprocess.run(cmd, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
|
||||
print(f"Created {TEST_FILE}")
|
||||
|
||||
# 2. Run Encoder in Debug Mode
|
||||
print("\nRunning Smart Encoder on test file...")
|
||||
print("(This should show detailed ffmpeg output)")
|
||||
print("="*60)
|
||||
|
||||
cmd = [
|
||||
"python", "smart_gpu_encoder.py",
|
||||
"--tv-dir", str(TEST_DIR.resolve()),
|
||||
"--content-dir", str(TEST_DIR.resolve()),
|
||||
"--jobs", "1",
|
||||
"--debug"
|
||||
]
|
||||
|
||||
subprocess.run(cmd)
|
||||
|
||||
print("\n--- TEST COMPLETE ---")
|
||||
print("Check for .mkv files in C:\\Users\\bnair\\Videos\\encodes")
|
||||
Reference in New Issue
Block a user