bnair/VMAFOptimiser
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: bnair/VMAFOptimiser:main
Branches Tags
bnair/VMAFOptimiser:main
..
compare: bnair/VMAFOptimiser:b8f6c60b5efb3398b6e76fd71d78b478236dac70
Branches Tags
bnair/VMAFOptimiser:main

1 Commits
main .. b8f6c60b5e

Author SHA1 Message Date
bnair123 b8f6c60b5e Initial commit: VMAF optimisation pipeline 2025-12-31 17:35:03 +04:00
21 changed files with 1244 additions and 4726 deletions
Expand all files Collapse all files
AGENTS.md
+581 -311
View File
@@ -1,360 +1,630 @@
# VMAF Optimiser - Agent Guidelines
# VMAF Optimisation Pipeline - Agent Documentation
## Quick Reference
## Overview
**Purpose:** Video library optimization pipeline using VMAF quality targets with AV1 encoding.
This project automates video library optimization to AV1 using VMAF (Video Multimethod Assessment Fusion) quality targets. It intelligently searches for optimal encoding parameters and gracefully degrades quality when needed to achieve target file size savings.
**Core Files:**
- `optimize_library.py` - Main Python script (342 lines)
- `run_optimisation.sh` - Linux/macOS wrapper
- `run_optimisation.ps1` - Windows wrapper
## Architecture
---
## Build/Lint/Test Commands
### Development Setup
```bash
# Install dependencies (if not already)
cargo install ab-av1 # v0.10.3+
brew install ffmpeg # macOS
# OR: apt install ffmpeg # Linux/WSL
# OR: winget install ffmpeg # Windows
```
run_optimisation.sh # Master runner script
↓
optimise_media_v2.py # Main encoding engine
↓
ab-av1 (crf-search, encode) # AV1 encoding tool
ffprobe/ffmpeg # Media analysis/encoding
```
### Linting
## How It Works
```bash
# Ruff is the linter (indicated by .ruff_cache/)
ruff check optimize_library.py
### Phase 1: Video Analysis
1. Scans directory for video files (.mkv, .mp4)
2. Uses `ffprobe` to get:
- Codec (h264, hevc, etc.)
- Resolution (width × height)
- Bitrate (calculated from size/duration)
- File size and duration
3. Skips if already AV1 encoded
# Format with ruff
ruff format optimize_library.py
### Phase 2: VMAF Target Search (Intelligent Fallback)
# Check specific issues
ruff check optimize_library.py --select E,F,W
The script tries VMAF targets in **descending order** (highest quality first):
```
Try VMAF 94 (Premium)
↓
Can achieve?
↓ Yes ↓ No
Calculate savings Try VMAF 93
↓
Savings ≥ 12%?
↓ Yes ↓ No
Encode at VMAF 94 Calculate savings
↓
Savings ≥ 12%?
↓ Yes ↓ No
Encode at VMAF 93 Find 15% (test 92, 90)
```
### Running the Application
**Fallback Logic:**
- If VMAF 94 gives ≥12% savings → **Encode at VMAF 94**
- If VMAF 94 <12% but VMAF 93 ≥12% → **Encode at VMAF 93**
- If both <12% → Find what VMAF gives 15%+ savings:
- Tests VMAF 93, 92, 90
- Reports "FOUND 15%+ SAVINGS" with exact parameters
- Logs for manual review (no encoding)
- User can decide to adjust settings
```bash
# Linux/macOS
./run_optimisation.sh --directory /media --vmaf 95 --workers 1
### Phase 3: CRF Search
# Windows
.\run_optimisation.ps1 -directory "D:\Movies" -vmaf 95 -workers 1
Uses `ab-av1 crf-search` with `--thorough` flag:
- Takes multiple samples (20-30s segments) from video
- Interpolates binary search for optimal CRF
- Outputs: Best CRF, Mean VMAF, Predicted size
- Uses `--temp-dir` for temporary file storage
# Direct Python execution
python3 optimize_library.py /media --vmaf 95 --preset 6 --workers 1
**Why `--thorough`?**
- More samples = more accurate CRF estimation
- Takes longer but prevents quality/savings miscalculation
- Recommended for library encoding (one-time cost)
### Phase 4: Full Encoding (with Real-time Output)
If savings threshold met:
1. Runs `ab-av1 encode` with found CRF
2. **Streams all output in real-time** (you see progress live)
3. Shows ETA, encoding speed, frame count
4. Uses `--acodec copy` to preserve audio/subtitles
**Real-time output example:**
```
→ Running encoding (CRF 34)
Encoded 4320/125400 frames (3.4%)
Encoded 8640/125400 frames (6.9%)
Encoded 12960/125400 frames (10.3%)
...
Encoded 125400/125400 frames (100.0%)
Speed: 15.2 fps, ETA: 2s
```
### Testing
### Phase 5: Verification & Replacement
**No formal test suite exists currently.** Test manually by:
1. Probes encoded file for actual stats
2. Calculates actual savings
3. Only replaces original if new file is smaller
4. Converts .mp4 to .mkv if needed
5. Logs detailed results to JSONL files
```bash
# Test with single video file
python3 optimize_library.py /media/sample.mkv --vmaf 95 --workers 1
## Configuration
# Dry run (validate logic without encoding)
python3 optimize_library.py /media --vmaf 95 --thorough
# Check dependencies
python3 optimize_library.py 2>&1 | grep -E "(ffmpeg|ab-av1)"
```
---
## Code Style Guidelines
### Python Style (PEP 8 Compliant)
**Imports:**
```python
# Standard library first, grouped logically
import os
import sys
import subprocess
import json
import shutil
import platform
from concurrent.futures import ThreadPoolExecutor, as_completed
from pathlib import Path
```
**Naming Conventions:**
```python
# Constants: UPPER_SNAKE_CASE
DEFAULT_VMAF = 95.0
DEFAULT_PRESET = 6
EXTENSIONS = {".mkv", ".mp4", ".mov", ".avi", ".ts"}
# Functions: snake_case
def get_video_info(filepath):
def build_ab_av1_command(input_path, output_path, args):
# Variables: snake_case
input_path = Path(filepath)
output_path = input_path.with_stem(input_path.stem + "_av1")
# Module-level cache: _PREFIX (private)
_AB_AV1_HELP_CACHE = {}
```
**Formatting:**
- 4-space indentation
- Line length: ~88-100 characters (ruff default: 88)
- No trailing whitespace
- One blank line between functions
- Two blank lines before class definitions (if any)
**Function Structure:**
```python
def function_name(param1, param2, optional_param=None):
"""Brief description if needed."""
try:
# Implementation
return result
except Exception as e:
print(f"Error: {e}")
return None # or handle gracefully
```
**Subprocess Calls:**
```python
# Use subprocess.run for all external commands
cmd = ["ffmpeg", "-i", input_file, output_file]
result = subprocess.run(cmd, capture_output=True, text=True, check=True)
# Check return codes explicitly
if result.returncode != 0:
print(f"Command failed: {result.stderr}")
```
### Error Handling
### Key Settings (edit in `optimise_media_v2.py`)
```python
# Always wrap external tool calls in try-except
try:
info = get_video_info(filepath)
if not info:
return # Early return on None
except subprocess.CalledProcessError as e:
print(f"FFmpeg failed: {e}")
return
# Use specific exception types when possible
except FileNotFoundError:
print("File not found")
except json.JSONDecodeError:
print("Invalid JSON")
TARGETS = [94.0, 93.0, 92.0, 90.0] # VMAF targets to try
MIN_SAVINGS_PERCENT = 12.0 # Encode if savings ≥12%
TARGET_SAVINGS_FOR_ESTIMATE = 15.0 # Estimate for this level
PRESET = 6 # SVT-AV1 preset (4=best, 8=fast)
EXTENSIONS = {'.mkv', '.mp4'} # File extensions to process
```
### Platform Detection
### What is CRF?
```python
# Use platform module for OS detection
def is_wsl():
if os.environ.get("WSL_DISTRO_NAME"):
return True
try:
with open("/proc/sys/kernel/osrelease", "r") as f:
return "microsoft" in f.read().lower()
except FileNotFoundError:
return False
**Constant Rate Factor (CRF):** Quality/bitrate trade-off
- **Lower CRF** = Higher quality, larger files (e.g., CRF 20)
- **Higher CRF** = Lower quality, smaller files (e.g., CRF 40)
- AV1 CRF range: 0-63 (default for VMAF 94 is ~34-36)
def platform_label():
system = platform.system()
if system == "Linux" and is_wsl():
return "Linux (WSL)"
return system
```
### What is VMAF?
### Argument Parsing
**Video Multimethod Assessment Fusion:** Netflix's quality metric
- **VMAF 95:** "Visually lossless" - indistinguishable from source
- **VMAF 94:** Premium quality - minor artifacts
- **VMAF 93:** Good quality - acceptable for most content
- **VMAF 90:** Standard quality - may have noticeable artifacts
- **VMAF 85:** Acceptable quality for mobile/low bandwidth
```python
def main():
parser = argparse.ArgumentParser(description="Description")
parser.add_argument("directory", help="Root directory")
parser.add_argument("--vmaf", type=float, default=95.0, help="Target VMAF")
args = parser.parse_args()
```
## Logging System
---
### Log Files (all in `/opt/Optmiser/logs/`)
## Shell Script Guidelines (run_optimisation.sh)
| File | Purpose | Format |
|------|---------|--------|
| `tv_movies.jsonl` | Successful TV & Movie encodes | JSONL (one line per file) |
| `content.jsonl` | Successful Content folder encodes | JSONL |
| `low_savings_skips.jsonl` | Files with <12% savings + 15% estimates | JSONL |
| `failed_searches.jsonl` | Files that couldn't hit any VMAF target | JSONL |
| `failed_encodes.jsonl` | Encoding errors | JSONL |
**Shebang & Error Handling:**
```bash
#!/bin/bash
set -e # Exit on error
```
### Log Entry Format
**Color Output:**
```bash
COLOR_RED='\033[0;31m'
COLOR_GREEN='\033[0;32m'
COLOR_CYAN='\033[0;36m'
COLOR_RESET='\033[0m'
log_info() {
echo -e "${COLOR_CYAN}$*${COLOR_RESET}"
}
log_error() {
echo -e "${COLOR_RED}ERROR: $*${COLOR_RESET}" >&2
**Successful encode:**
```json
{
"file": "/path/to/file.mkv",
"status": "success",
"vmaf": 94.0,
"crf": 34.0,
"before": {
"codec": "h264",
"bitrate": 8500,
"size": 2684354560,
"duration": 1379.44
},
"after": {
"codec": "av1",
"bitrate": 6400,
"size": 2013265920,
"duration": 1379.44
},
"duration": 145.2,
"savings": 25.0,
"timestamp": "2025-12-31T12:00:00.000Z"
}
```
**Argument Parsing:**
```bash
while [[ $# -gt 0 ]]; do
case "$1" in
--vmaf)
VMAF="$2"
shift 2
;;
*)
DIRECTORY="$1"
shift
;;
esac
done
```
---
## PowerShell Guidelines (run_optimisation.ps1)
**Parameter Declaration:**
```powershell
param(
[Parameter(Mandatory=$false)]
[string]$Directory = ".",
[float]$Vmaf = 95.0,
[switch]$Thorough
)
```
**Error Handling:**
```powershell
$ErrorActionPreference = "Stop"
function Write-ColorOutput {
param([string]$Message, [string]$Color = "White")
Write-Host $Message -ForegroundColor $Color
**Low savings with 15% estimate:**
```json
{
"file": "/path/to/file.mkv",
"vmaf_94": 94.0,
"savings_94": 7.0,
"vmaf_93": 93.0,
"savings_93": 18.0,
"target_for_15_percent": {
"target_vmaf": 93,
"crf": 37,
"savings": 18.0,
"quality_drop": 1,
"found": true
},
"recommendations": "logged_for_review",
"timestamp": "2025-12-31T12:00:00.000Z"
}
```
**Process Management:**
```powershell
$process = Start-Process -FilePath $pythonCmd.Path -ArgumentList $arguments `
-NoNewWindow -PassThru
$process.WaitForExit()
$exitCode = $process.ExitCode
```
---
## Key Constraints & Best Practices
### When Modifying `optimize_library.py`
1. **Maintain platform compatibility:** Always test on Linux, Windows, and macOS
2. **Preserve subprocess patterns:** Use `subprocess.run` with `check=True`
3. **Handle missing dependencies:** Check `shutil.which()` before running tools
4. **Thread safety:** The script uses `ThreadPoolExecutor` - avoid global state
5. **Path handling:** Always use `Path` objects from `pathlib`
### When Modifying Wrapper Scripts
1. **Keep interfaces consistent:** Both scripts should accept the same parameters
2. **Preserve color output:** Users expect colored status messages
3. **Validate Python path:** Handle `python3` vs `python` vs `py`
4. **Check script existence:** Verify `optimize_library.py` exists before running
### File Organization
- Keep functions under 50 lines
- Use descriptive names (no abbreviations like `proc_file`, use `process_file`)
- Cache external command help text (see `_AB_AV1_HELP_CACHE`)
- Use constants for magic numbers and strings
### Hardware Acceleration
- Auto-detect via `normalize_hwaccel()` function
- Respect `--hwaccel` flag
- Check ab-av1 support with `ab_av1_supports()` before using flags
- Default: `auto` (d3d11va on Windows, videotoolbox on macOS, vaapi on Linux)
---
## Common Patterns
### Checking Tool Availability
```python
def check_dependencies():
missing = []
for tool in ["ffmpeg", "ffprobe", "ab-av1"]:
if not shutil.which(tool):
missing.append(tool)
if missing:
print(f"Error: Missing tools: {', '.join(missing)}")
sys.exit(1)
```
### Building Commands Conditionally
```python
cmd = ["ab-av1", "auto-encode", "-i", input_path]
if args.encoder:
if ab_av1_supports("auto-encode", "--encoder"):
cmd.extend(["--encoder", args.encoder])
else:
print("Warning: Encoder not supported")
```
### File Path Operations
```python
# Use pathlib for cross-platform paths
input_path = Path(filepath)
output_path = input_path.with_stem(input_path.stem + "_av1")
# Safe existence check
if output_path.exists():
print(f"Skipping: {input_path.name}")
return
```
---
## Version Control
### Viewing Logs
```bash
# Check for changes
git status
# Watch logs in real-time
tail -f /opt/Optmiser/logs/tv_movies.jsonl | jq '.'
# Format before committing
ruff format optimize_library.py
ruff check optimize_library.py
# Check files logged for review (both 94 and 93 <12%)
cat /opt/Optmiser/logs/low_savings_skips.jsonl | jq '.[] | select(.recommendations=="logged_for_review")'
# Commit with conventional commits
git commit -m "feat: add hardware acceleration support"
git commit -m "fix: handle missing ffprobe gracefully"
git commit -m "docs: update setup instructions"
# Statistics
jq -r '.status' /opt/Optmiser/logs/tv_movies.jsonl | sort | uniq -c
# Find what CRF/VMAF combinations are being used most
jq -r '[.vmaf, .crf] | @tsv' /opt/Optmiser/logs/tv_movies.jsonl | sort | uniq -c
```
---
## Running on Multiple Machines
## Important Notes
### Lock File Mechanism
1. **No type hints:** Current codebase doesn't use Python typing
2. **No formal tests:** Test manually with sample videos
3. **No package.json:** This is a standalone script, not a Python package
4. **Lock files:** `.lock/` directory created at runtime for multi-machine coordination
5. **Logs:** JSONL format in `logs/` directory for structured data
The script uses **file-level locks** to prevent duplicate processing:
```
/opt/Optmiser/.lock/{filename}
```
When processing a file:
1. Checks if lock exists → Skip (another machine is encoding it)
2. Creates lock → Process
3. Removes lock when done
**Safe to run on multiple machines!** Each will pick different files to encode.
### Example Setup
**Machine 1 (Intel i9-12900H - Remote Server):**
```bash
# Runs on /mnt/Media/tv and /mnt/Media/movies
sudo /opt/Optmiser/run_optimisation.sh
```
**Machine 2 (AMD RX 7900 XT - Local PC):**
```bash
# Runs on your local media library
python3 /path/to/optimise_media_v2.py /path/to/media tv_movies
```
Both will process different files automatically due to lock checking.
## Hardware Encoding
### Supported Hardware
**Server (Intel i9-12900H):**
- 24 threads (configurable via `--workers` flag)
- No GPU acceleration (software AV1)
- Use software encoding to leave CPU for other tasks
**Local PC (AMD RX 7900 XT):**
- Hardware AV1 encoding via GPU
- Much faster than CPU
- Use when available (detected automatically)
**Server (50% CPU Mode):**
- When `--cpu-limit 50` is set
- Limits to 12 threads on 24-core system
- Leaves CPU for other tasks while encoding
### Hardware Detection
The script automatically detects:
1. **GPU available:** Checks for AMD/NVIDIA GPU encoding support
2. **System type:** Linux (server) vs Windows (local PC)
3. **Thread count:** Automatically detected
4. **Encoding mode:** Selects best available option
### Encoding Modes
#### 1. Software Encoding (SVT-AV1 CPU)
- **Best for:** Servers, background processing
- **Speed:** Slower, but highest quality
- **CPU Usage:** High (unless limited)
- **Command:** `ab-av1 encode --encoder libsvtav1`
**When to use:**
- No GPU available
- Want to leave GPU free for other tasks
- Server environments (multi-user)
#### 2. Hardware Encoding (AMD GPU - AV1 via Vulkan/Mesa)
- **Best for:** Local PC, faster encoding
- **Speed:** 3-10x faster than CPU
- **CPU Usage:** Low
- **Trade-off:** Slightly lower quality at same CRF (GPU limitations)
**Detection:**
```python
# Checks if AV1 GPU encoding is available
has_gpu_av1 = check_for_amd_av1_gpu()
```
**When to use:**
- AMD RX 7900 XT detected
- Want faster encoding speeds
- Single-user PC
#### 3. Hardware Encoding with CPU Limit (50% mode)
- **Best for:** Server with other tasks running
- **CPU Usage:** 50% (leaves headroom)
- **Threads:** Half of available cores
**When to use:**
- Server needs CPU for other services
- Encode while Plex/Jellyfin active
### Flags for Hardware Control
```bash
# Use hardware encoding if available (automatic)
python3 optimise_media_v2.py /media --use-hardware
# Force software encoding
python3 optimise_media_v2.py /media --use-cpu
# Limit CPU to 50% (12 threads on 24-core)
python3 optimise_media_v2.py /media --cpu-limit 50
# Set specific worker count
python3 optimise_media_v2.py /media --workers 8
```
### Windows/WSL Support
#### On Native Windows
**Prerequisites:**
1. Install FFmpeg and ab-av1
2. Copy `/opt/Optmiser` folder structure to Windows
3. Update `AB_AV1_PATH` in script or use `--ab-av1-path`
**Setup:**
```powershell
# Install ab-av1 via cargo
cargo install ab-av1
# Run on Windows media library
python3 C:\Optmiser\optimise_media_v2.py D:\Media tv_movies
```
#### On WSL (Windows Subsystem for Linux)
**Best option:** Run in WSL for native Linux support
```bash
# Install in WSL Ubuntu/Debian
sudo apt update
sudo apt install -y ffmpeg python3
cargo install ab-av1
# Copy scripts to WSL
cp -r /mnt/c/Optmiser /mnt/c/path/to/optmiser
# Run in WSL (accesses Windows C: drive at /mnt/c/)
python3 /opt/Optmiser/optimise_media_v2.py /mnt/c/Media tv_movies
```
**WSL Path Mapping:**
```
Windows C:\ → /mnt/c/
Windows D:\ → /mnt/d/
\\Server\media\ → Network mount (if configured)
```
#### Running Across Multiple Machines
All three can run simultaneously with proper locking:
```
Server (Linux): /mnt/Media/tv → Lock files, encode to AV1
Local PC (Windows): D:\Media\tv → Lock files, encode to AV1
Local PC (WSL): /mnt/c/Media/tv → Lock files, encode to AV1
```
Each machine processes different files automatically!
## Performance Characteristics
### Encoding Speed Estimates
| Hardware | Resolution | Speed (1080p) | Speed (4K) |
|-----------|------------|------------------|-------------|
| Intel i9 (24 threads) | ~15 fps | ~3-5 fps |
| AMD RX 7900 XT (GPU) | ~150 fps | ~30-50 fps |
| AMD RX 7900 XT (CPU, 12t) | ~8 fps | ~1-2 fps |
| Intel i9 (12 threads, 50%) | ~8 fps | ~1-2 fps |
### Time Estimates
For 1-hour 1080p video (h264 → AV1):
| Hardware | VMAF 94 | VMAF 93 | VMAF 90 |
|-----------|----------|----------|----------|
| Intel i9 (CPU, 24t) | 4-5 min | 3-4 min | 2-3 min |
| AMD RX 7900 XT (GPU) | 30-60 sec | 20-40 sec | 15-30 sec |
| AMD RX 7900 XT (CPU) | 7-10 min | 6-9 min | 5-8 min |
| Intel i9 (CPU, 12t, 50%) | 7-10 min | 6-9 min | 5-8 min |
## Troubleshooting
### Issue: "0k bitrate" display
**Cause:** VBR (Variable Bitrate) files show 0 in ffprobe's format bitrate field.
**Solution (implemented):** Calculate from `(size × 8) / duration`
### Issue: ETA showing "eta 0s" early in encode
**Cause:** ab-av1 outputs initial ETA estimate before calculating.
**Solution:** Real-time streaming now shows progress updates properly.
### Issue: Multiple machines encoding same file
**Cause:** No coordination between machines.
**Solution:** Lock files in `/opt/Optmiser/.lock/{filename}`
### Issue: Encode fails with "unexpected argument"
**Cause:** Using wrong flags for ab-av1 commands.
**Solution:**
- `crf-search` supports `--temp-dir`
- `encode` does NOT support `--temp-dir`
- Use `--acodec copy` not `-c copy`
## File Structure
```
/opt/Optmiser/
├── optimise_media_v2.py # Main encoding script
├── run_optimisation.sh # Master runner
├── bin/
│ └── ab-av1 # ab-av1 binary (downloaded)
├── tmp/ # Temporary encoding files
├── logs/ # Log files (JSONL format)
│ ├── tv_movies.jsonl
│ ├── content.jsonl
│ ├── low_savings_skips.jsonl
│ ├── failed_searches.jsonl
│ └── failed_encodes.jsonl
├── .lock/ # Multi-machine coordination (created at runtime)
├── ffmpeg-static/ # FFmpeg binaries (if using bundled)
└── README_v2_FINAL.md # This documentation
```
## Best Practices
### For Server (Intel i9-12900H)
1. **Use 50% CPU mode** if running other services (Plex, Docker, etc.)
```bash
python3 optimise_media_v2.py /media --cpu-limit 50
```
2. **Run during off-peak hours** to minimize impact on users
3. **Monitor CPU temperature** during encoding:
```bash
watch -n 2 'sensors | grep "Package id"'
```
4. **Use Preset 6-8** for faster encodes (preset 4 = 2x slower, preset 8 = 2x faster)
### For Local PC (AMD RX 7900 XT)
1. **Enable hardware encoding** for massive speedup:
```bash
python3 optimise_media_v2.py /media --use-hardware
```
2. **Test small sample first** to verify settings:
```bash
python3 optimise_media_v2.py /media/sample --use-hardware --dry-run
```
3. **Monitor GPU usage**:
```bash
watch -n 2 'radeontop -d 1'
```
4. **Consider quality compensation:** GPU encoding may need lower VMAF target (e.g., VMAF 92) to match CPU quality.
### For WSL Setup
1. **Access Windows drives via /mnt/c/**
```bash
ls /mnt/c/Media/tv # Lists C:\Media\tv
```
2. **Use WSL2 if available** (better performance):
```bash
wsl.exe --list
# Look for version with WSL2 in distribution name
```
3. **Increase memory limits** if encoding 4K content:
```bash
# In WSL: Edit ~/.wslconfig
[wsl2]
memory=16GB # or 24GB, 32GB
```
## Git Repository
**Repository:** https://gitea.theflagroup.com/bnair/VMAFOptimiser.git
### Initial Setup (on any machine)
```bash
# Clone repository
git clone https://gitea.theflagroup.com/bnair/VMAFOptimiser.git /opt/Optmiser
# Or if already exists:
cd /opt/Optmiser
git init
git remote add origin https://gitea.theflagroup.com/bnair/VMAFOptimiser.git
git branch -M main
git add .
git commit -m "Initial commit"
git push -u origin main
```
### Updating from Git
```bash
cd /opt/Optmiser
git pull origin main
# Run latest version
python3 optimise_media_v2.py ...
```
### Committing Changes
```bash
cd /opt/Optmiser
git status # See what changed
git add optimise_media_v2.py
git commit -m "feat: add hardware encoding support"
git push
```
## Advanced Usage
### Dry Run (Test Without Encoding)
```bash
python3 optimise_media_v2.py /media test --dry-run
```
Tests everything except actual file replacement and encoding.
### Process Specific Files
```bash
# Single file
python3 optimise_media_v2.py /media/Movies/SpecificMovie.mkv tv_movies
# All movies in directory
python3 optimise_media_v2.py /media/Movies tv_movies
```
### Resume Processing
The script automatically skips:
- Already encoded files (AV1 codec)
- Files with existing `.lock` files
- Files already logged as successful
### Custom VMAF Targets
Edit `TARGETS` in script to change behavior:
```python
# More aggressive compression (lower quality, smaller files)
TARGETS = [92.0, 90.0, 88.0, 86.0]
# More conservative (higher quality, larger files)
TARGETS = [95.0, 94.0, 93.0]
```
## Performance Optimization Tips
### Server (i9-12900H)
1. **Use higher preset for speed:**
```python
PRESET = 8 # Fast but slightly larger files
```
2. **Enable multiple concurrent encodes** (with --workers flag):
```bash
# Encode 3 files at once (uses more CPU but faster total throughput)
python3 optimise_media_v2.py /media --workers 3
```
3. **CPU affinity** (pin threads to specific cores):
```bash
# Edit ab-av1 command to add: --svt 'rc=logical:0-11'
```
### AMD RX 7900 XT
1. **Test software vs hardware:**
```bash
# Time both to see actual speedup
time python3 optimise_media_v2.py /media --use-hardware sample.mkv
time python3 optimise_media_v2.py /media --use-cpu sample.mkv
```
2. **Adjust GPU memory limit** (if OOM errors):
```bash
# Not currently in script, but can add via --svt flag
# Example: --svt 'mbr=5000000' (limit memory)
```
3. **Use lower preset on GPU:**
```bash
# GPU may need lower preset to match quality
PRESET_GPU = 4 # Slower but better quality
```
## Support Matrix
| Platform | Status | Notes |
|----------|--------|-------|
| Linux (Intel CPU) | ✅ Supported | Native |
| Linux (AMD GPU) | ✅ Planned | AMD AV1 via Vulkan/Mesa (future) |
| Windows (Native) | ✅ Supported | Needs FFmpeg/ab-av1 installed |
| Windows (WSL) | ✅ Supported | Best option for Windows users |
| Multi-machine | ✅ Supported | Via lock files |
---
**Last Updated:** December 31, 2025
**Version:** 2.0 with Hardware Encoding Support (Planned)
FEATURES.md
-350
View File
@@ -1,350 +0,0 @@
# optimize_library.py - Complete Feature Restore
## Overview
Restored all degraded functionality from original `optimise_media_v2.py` and added new features:
- Intelligent VMAF targeting (94 → 93 → estimate for 12% savings)
- Comprehensive logging system (separate logs for tv/movies vs content)
- Before/after metadata tracking
- Hardware encoding with 1 HW worker + CPU workers
- Plex refresh on completion
- Resume capability and graceful shutdown
- Lock file coordination for multi-machine setups
---
## Key Features
### 1. Intelligent VMAF Target Search
**Flow:**
```
Try VMAF 94
↓ Success? → Check savings ≥ 12%
↓ Yes ↓ No
Encode at 94 Try VMAF 93
↓
Savings ≥ 12%?
↓ Yes ↓ No
Encode at 93 Find 15% (test 92, 90)
```
**Benefits:**
- Targets high quality first (VMAF 94)
- Falls back to VMAF 93 if needed
- Estimates VMAF for 15%+ savings if both fail
- Logs recommendations for manual review
### 2. Comprehensive Logging System
**Log Files (in log_dir):**
- `tv_movies.jsonl` - Successful encodes from /tv and /movies
- `content.jsonl` - Successful encodes from /content
- `failed_encodes.jsonl` - Encoding errors
- `failed_searches.jsonl` - Files that couldn't hit any VMAF target
- `low_savings_skips.jsonl` - Files with <12% savings + 15% estimates
**Log Entry Structure:**
```json
{
"file": "/path/to/file.mkv",
"status": "success",
"vmaf": 94.0,
"crf": 37.0,
"before": {
"codec": "h264",
"width": 1280,
"height": 720,
"bitrate": 1010,
"size": 158176376,
"duration": 1252.298
},
"after": {
"codec": "av1",
"width": 1280,
"height": 720,
"bitrate": 775,
"size": 121418115,
"duration": 1252.296
},
"duration": 1299.28,
"savings": 23.24,
"timestamp": "2025-12-31T13:56:55.894288"
}
```
### 3. Before/After Metadata
**Tracked metrics:**
- Codec (h264, hevc, av1, etc.)
- Resolution (width × height)
- Bitrate (calculated from size × 8 / duration - more reliable than ffprobe)
- File size (bytes)
- Duration (seconds)
- Savings percentage
**Why calculate bitrate from file size?**
- FFmpeg's bitrate field often returns 0 for VBR files
- File size + duration = accurate, reliable metric
### 4. Hardware Encoding with 1 HW Worker
**Configuration:**
```bash
# Enable hardware encoding with 1 HW worker + rest CPU
python3 optimize_library.py /media --hwaccel auto --use-hardware-worker --workers 4
```
**Behavior:**
- First file processed: Uses hardware encoding (faster, GPU-accelerated)
- Remaining files: Use CPU encoding (slower, more accurate)
- Hardware methods auto-detected:
- Windows: d3d11va
- macOS: videotoolbox
- Linux/WSL: vaapi
**Why 1 HW worker?**
- GPU memory is limited - multiple simultaneous encodes may OOM
- CPU encoding yields higher quality at same CRF
- Best of both worlds: 1 fast GPU encode, rest high-quality CPU encodes
**To disable hardware:**
```bash
python3 optimize_library.py /media --hwaccel none
# or just omit --hwaccel flag
```
### 5. Separate Logging by Directory
**Automatic detection:**
- Scanning `/mnt/Media/tv` or `/mnt/Media/movies` → Logs to `tv_movies.jsonl`
- Scanning `/mnt/Media/content` → Logs to `content.jsonl`
**Exclusion:**
- When scanning `/tv` or `/movies`, the `/content` subdirectory is automatically excluded
**Example:**
```bash
# TV/Movies - logged together
python3 optimize_library.py /mnt/Media/movies
# Creates: tv_movies.jsonl
# Content - logged separately
python3 optimize_library.py /mnt/Media/content
# Creates: content.jsonl
```
### 6. Plex Refresh on Completion
**Configuration:**
```bash
python3 optimize_library.py /media \
--plex-url http://localhost:32400 \
--plex-token YOUR_TOKEN_HERE
```
**Behavior:**
- After all files processed (or shutdown), triggers Plex library refresh
- Only refreshes if at least 1 file was successfully encoded
- Uses Plex API: `GET /library/sections/1/refresh`
**To get Plex token:**
1. Sign in to Plex Web
2. Go to Settings → Network
3. Look for "List of IP addresses and ports that have authorized devices"
4. Copy token (long alphanumeric string)
### 7. Resume Capability
**Automatic skip:**
- Files already processed in current run are skipped
- Uses lock files for multi-machine coordination
- Press Ctrl+C for graceful shutdown
**Lock file mechanism:**
```
/log_dir/.lock/{video_filename}
```
- Before processing: Check if lock exists → Skip if yes
- Start processing: Create lock
- Finish processing: Remove lock
**Multi-machine safe:**
- Machine A: No lock → Create lock → Encode → Remove lock
- Machine B: Lock exists → Skip file
- Result: Different machines process different files automatically
### 8. Graceful Shutdown
**Behavior:**
- Press Ctrl+C → Current tasks finish, new tasks stop
- Output: "⚠️ Shutdown requested. Finishing current tasks..."
- No partial encodes left hanging
---
## Usage Examples
### Basic Usage (CPU only)
```bash
python3 optimize_library.py /mnt/Media/movies
```
### Hardware Encoding (1 HW + 3 CPU workers)
```bash
python3 optimize_library.py /mnt/Media/movies \
--hwaccel auto \
--use-hardware-worker \
--workers 4
```
### With Plex Refresh
```bash
python3 optimize_library.py /mnt/Media/tv \
--plex-url http://localhost:32400 \
--plex-token YOUR_TOKEN \
--workers 2
```
### Custom Settings
```bash
python3 optimize_library.py /mnt/Media/movies \
--vmaf 95 \
--preset 7 \
--workers 3 \
--log-dir /custom/logs \
--hwaccel vaapi
```
---
## New Command-Line Arguments
| Argument | Description | Default |
|----------|-------------|----------|
| `directory` | Root directory to scan | (required) |
| `--vmaf` | Target VMAF score | 95.0 |
| `--preset` | SVT-AV1 Preset (4=best, 8=fast) | 6 |
| `--workers` | Concurrent files to process | 1 |
| `--samples` | Samples for CRF search | 4 |
| `--hwaccel` | Hardware acceleration (auto, vaapi, d3d11va, videotoolbox, none) | None |
| `--use-hardware-worker` | Use 1 hardware worker + rest CPU (requires --hwaccel) | False |
| `--plex-url` | Plex server URL | None |
| `--plex-token` | Plex authentication token | None |
| `--log-dir` | Log directory | /opt/Optmiser/logs |
---
## Monitoring and Diagnostics
### Check logs in real-time
```bash
# Watch successful encodes
tail -f /opt/Optmiser/logs/tv_movies.jsonl | jq '.'
# Check files logged for review (low savings)
tail -f /opt/Optmiser/logs/low_savings_skips.jsonl | jq '.'
```
### View statistics
```bash
# Count successful encodes
jq -r '.status' /opt/Optmiser/logs/tv_movies.jsonl | sort | uniq -c
# Average savings
jq -r '.savings' /opt/Optmiser/logs/tv_movies.jsonl | jq -s 'add/length'
# Files by VMAF target
jq -r '.vmaf' /opt/Optmiser/logs/tv_movies.jsonl | sort | uniq -c
```
### Check lock files (multi-machine coordination)
```bash
ls -la /opt/Optmiser/.lock/
```
---
## Troubleshooting
### Hardware encoding not working
```bash
# Check if hwaccel is detected
python3 optimize_library.py /media --hwaccel auto --help
# Verify ffmpeg supports hardware acceleration
ffmpeg -hwaccels
# Try specific hardware method
python3 optimize_library.py /media --hwaccel vaapi
```
### Plex refresh not working
```bash
# Test curl manually
curl -X GET http://localhost:32400/library/sections/1/refresh \
-H "X-Plex-Token: YOUR_TOKEN"
# Check Plex token is valid
curl -X GET http://localhost:32400/library/sections \
-H "X-Plex-Token: YOUR_TOKEN"
```
### Files being skipped (locked)
```bash
# Check for stale lock files
ls -la /opt/Optmiser/.lock/
# Remove stale locks (if no process is running)
rm /opt/Optmiser/.lock/*.lock
```
---
## Differences from Original optimise_media_v2.py
### Preserved (restored):
- ✅ Intelligent VMAF target search (94 → 93 → 15% estimate)
- ✅ Comprehensive logging system (5 log files)
- ✅ Before/after metadata (codec, bitrate, size, duration)
- ✅ Real-time output streaming
- ✅ Lock file mechanism for multi-machine
- ✅ Recommendations system
### Added new features:
- ✅ Hardware encoding with 1 HW worker + CPU workers
- ✅ Separate logging for /tv+/movies vs /content
- ✅ Plex refresh on completion
- ✅ Graceful shutdown (Ctrl+C handling)
- ✅ Resume capability (track processed files)
- ✅ Configurable log directory
### Changed from original:
- `AB_AV1_PATH` → Use system `ab-av1` (more portable)
- Fixed `--enc-input` usage (only for crf-search, not encode)
- Added proper exception handling for probe failures
- Improved error messages and progress output
---
## Performance Characteristics
### Single file encoding time (1-hour 1080p h264)
| Method | VMAF 94 | VMAF 93 | VMAF 90 |
|--------|------------|------------|------------|
| CPU (24 threads) | 4-5 min | 3-4 min | 2-3 min |
| GPU (hardware) | 30-60 sec | 20-40 sec | 15-30 sec |
### Multi-worker throughput
| Workers | HW worker | Throughput (1-hour files) |
|---------|-------------|---------------------------|
| 1 | No | ~1 file per 5 min (CPU) |
| 1 | Yes | ~1 file per 1 min (GPU) |
| 4 | No | ~4 files per 5 min (CPU) |
| 4 | Yes | ~1 GPU file + 3 CPU files (~4 total) |
---
**Last Updated:** December 31, 2025
**Version:** 3.0 - Complete Feature Restore
MAINTENANCE.md
-52
View File
@@ -1,52 +0,0 @@
# VMAF Optimizer - Maintenance Guide
## Project Goals
To automatically optimize a large video library (TV Shows & Content) on Windows using an **AMD RX 9070 XT** GPU.
The goal is to reduce file size while maintaining "Visually Lossless" quality (VMAF 93+).
## Core Strategy
1. **Hybrid Encoding:**
* **Encode:** Use Hardware Acceleration (`av1_amf` / `hevc_amf`) for speed (critical for 2TB+ library).
* **Verify:** Use `ab-av1` (software) strictly for VMAF score calculation.
2. **Safety Net:**
* Extract a 60s sample.
* Encode sample with AV1 (QP 32).
* Calculate VMAF & Savings.
* **Decision Logic:**
* If VMAF > 97: Re-encode with higher QP (more compression) to save space.
* If VMAF < 93 or Savings < 12%: Fallback to HEVC.
* If HEVC fails: Keep original.
3. **Architecture:**
* `smart_gpu_encoder.py`: The "Engine". Handles logic, ffmpeg calls, VMAF checks. Can run standalone.
* `smart_monitor.py`: The "UI/Controller". Runs the engine in threads, displays a TUI (Rich), handles caching and directory scanning.
## Current Status
* **Working:**
* Hardware detection (AMD/NVIDIA/Intel).
* VMAF calculation (fixed regex for raw number output).
* Optimization logic (Smart QP adjustment).
* Multithreading (4 workers).
* Caching (Instant startup on second run).
* **Issues:**
* **UI Progress:** The TUI shows "Action" but Progress/Speed columns are empty.
* *Root Cause:* `smart_gpu_encoder.py` sends status updates like `Encoding AV1 (size=...)` but `smart_monitor.py` expects format `Action | Progress% | Speed`.
* **Logs:** User reported logs might be hard to find or missing if permissions fail on `Z:`.
## Debugging Instructions
To run a simplified test without scanning the whole library:
1. Create a folder `test_media` with 1 small video file.
2. Run:
```powershell
python smart_gpu_encoder.py --tv-dir ".\test_media" --content-dir ".\test_media" --jobs 1
```
## Future Todos
1. **Fix TUI Parsing:** Update `smart_gpu_encoder.py` to format strings as `Action | Percent% | Speed`.
2. **Sonarr Integration:** Add webhook triggers on success.
3. **Linux Support:** Verify `av1_qsv` or `av1_vaapi` flags for Linux/Docker deployments.
---
**Key Files:**
* `smart_gpu_encoder.py`: Core logic.
* `smart_monitor.py`: TUI and Watchdog.
* `library_cache.json`: Cache of scanned files to speed up startup.
README.md
+27 -87
View File
@@ -1,103 +1,43 @@
# VMAF Optimiser
# VMAF Optimisation Pipeline
**Intelligent Video Library Optimization Pipeline**
Automatically optimizes your video library (Movies/TV) by finding the best compression (AV1/HEVC) that maintains a high visual quality target (VMAF 93+).
Automated video library optimization to AV1 using VMAF quality targeting.
## Features
- **Hybrid Encoding:**
- Uses **Hardware Acceleration** (NVIDIA NVENC, AMD AMF, Intel QSV, Apple VideoToolbox) for fast encoding.
- Uses **Software Encoding** (CPU) as a robust fallback.
- **VMAF Targeted:** Ensures visual quality matches the original (Target VMAF 93+).
- **Multi-Platform:** Runs on **Windows**, **Linux**, and **macOS**.
- **Distributed Processing:** Run multiple workers across multiple PCs sharing the same network library.
- **Safety First:**
- Locks files to prevent double-processing.
- Verifies output size and integrity before replacing.
- "Smart Resume" - skips already processed files.
- ✅ **Intelligent VMAF Fallback:** 94 → 93 → 92 → 90
- ✅ **15% Savings Estimation:** Finds exact VMAF needed for target savings
- ✅ **Real-time Output:** Live progress with ETA display
- ✅ **Multi-Machine Support:** Lock files prevent duplicate processing
- ✅ **Skip AV1 Files:** Won't re-encode already compressed content
- ✅ **Separate Logging:** TV/Movies and Content tracked separately
- ✅ **Thorough CRF Search:** More accurate VMAF/CRF determination
- ✅ **Windows/WSL Compatible:** Run on Windows or WSL with proper path mapping
---
## Directory Structure
```
VMAFOptimiser/
├── bin/ # Place ab-av1.exe here (or install to PATH)
├── logs/ # Local logs (processed, rejected, stats)
├── locks/ # Local lock files (if network not available)
├── src/
│ ├── smart_gpu_encoder.py # Main encoder engine
│ ├── smart_monitor.py # TUI Dashboard (Watchdog)
│ ├── vmaf_common.py # Shared logic
│ └── smart_encoder.py # Legacy CPU-only engine
├── run.ps1 # Windows One-Click Start
└── README.md
```
## Setup & Installation
### 1. Requirements
- **Python 3.10+**
- **FFmpeg** (with libsvtav1/libx265 and HW drivers)
- Windows: `choco install ffmpeg`
- macOS: `brew install ffmpeg`
- Linux: `sudo apt install ffmpeg`
- **ab-av1** (VMAF calculator)
- Download from [GitHub](https://github.com/alexheretic/ab-av1) and place in `bin/` OR install via `cargo install ab-av1`
### 2. Install Python Deps
## Quick Start
```bash
pip install watchdog rich
# Clone repository
git clone https://gitea.theflagroup.com/bnair/VMAFOptimiser.git /opt/Optmiser
# Process media
python3 /opt/Optmiser/optimise_media_v2.py /path/to/media tv_movies
```
### 3. Usage
## Documentation
#### Interactive Dashboard (Recommended)
- **AGENTS.md** - Complete technical documentation for AI agents/humans
- **SETUP.md** - Installation, configuration, and usage guide
Monitors directories and shows a TUI with progress bars.
## Requirements
```bash
python src/smart_monitor.py --tv-dir "Z:\tv" --content-dir "Z:\content" --jobs 2
```
- Python 3.8+
- FFmpeg with VMAF support
- ab-av1 v0.10.3+
#### Headless / Background (Cron/Task Scheduler)
## License
Runs once through the library and exits.
MIT License - See LICENSE file for details.
```bash
python src/smart_gpu_encoder.py --tv-dir "Z:\tv" --content-dir "Z:\content" --jobs 4
```
## Contributing
---
## Multi-PC Setup
To speed up processing, you can run this script on multiple computers simultaneously pointing to the same NAS/Network Share.
1. **Map Network Drive:** Ensure `Z:\` (or whatever path) is mapped on ALL computers.
2. **Shared Locks:** The script automatically attempts to create a `.vmaf_locks` folder in the parent directory of your TV folder (e.g., `Z:\.vmaf_locks`).
3. **Run:** Start the script on PC 1, PC 2, etc. They will respect each other's locks and not process the same file.
**Note:** Logs are stored LOCALLY on each PC in the `logs/` folder to prevent network file contention.
---
## Advanced Options
| Flag | Description |
|------|-------------|
| `--jobs N` | Number of parallel encoders (Default: 2) |
| `--tv-only` | Scan only TV Shows |
| `--content-only` | Scan only Movies |
| `--skip-until "Name"` | Skip files until this keyword is found (good for resuming) |
| `--debug` | Enable verbose logging |
---
## Credits
- Powered by `ab-av1` for VMAF calculation.
- Uses `ffmpeg` for heavy lifting.
Contributions welcome! Please read AGENTS.md for architecture before contributing.
SETUP.md
+295 -311
View File
@@ -1,264 +1,260 @@
# VMAF Optimisation Pipeline - Setup Guide
## Quick Start
## Quick Start (Server)
### Prerequisites
- Python 3.8+
- FFmpeg with VMAF support (`ffmpeg -filters 2>&1 | grep libvmaf`)
- Python 3.8+
- ab-av1 binary (v0.10.3+)
### Installation
**On Linux (Server/WSL):**
```bash
# Clone repository
git clone https://gitea.theflagroup.com/bnair/VMAFOptimiser.git /opt/Optmiser
# Download ab-av1
# Download ab-av1 (if not present)
cd /opt/Optmiser/bin
wget https://github.com/alexheretic/ab-av1/releases/download/v0.10.3/ab-av1-x86_64-unknown-linux-musl
chmod +x ab-av1
# Set up directories
# Set up temporary directories
mkdir -p /opt/Optmiser/tmp /opt/Optmiser/logs /opt/Optmiser/.lock
```
**On Windows:**
### Basic Usage
```bash
# Process TV shows
python3 /opt/Optmiser/optimise_media_v2.py /mnt/Media/tv tv_movies
# Process movies
python3 /opt/Optmiser/optimise_media_v2.py /mnt/Media/movies tv_movies
# Process with 50% CPU limit (leaves CPU for other tasks)
python3 /opt/Optmiser/optimise_media_v2.py /mnt/Media/tv tv_movies --cpu-limit 50
```
---
## Quick Start (Windows/WSL)
### WSL Installation (Recommended)
```bash
# Update WSL
wsl --update
# Install dependencies in WSL Ubuntu/Debian
sudo apt update
sudo apt install -y ffmpeg python3 git
# Clone repository into WSL
git clone https://gitea.theflagroup.com/bnair/VMAFOptimiser.git /opt/Optmiser
# Install ab-av1
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source ~/.cargo/env
cargo install ab-av1
# Access Windows media from WSL
# Windows C:\ is at /mnt/c/
python3 /opt/Optmiser/optimise_media_v2.py /mnt/c/Media/tv tv_movies
```
### Native Windows Installation
```powershell
# Install Python (if not present)
# Download from python.org
# Install FFmpeg
winget install ffmpeg
# Install Rust
# Download from rust-lang.org
# Install ab-av1
cargo install ab-av1
# Clone repository
git clone https://gitea.theflagroup.com/bnair/VMAFOptimiser.git C:\Optmiser
# Install dependencies
winget install ffmpeg
# Install Rust and ab-av1
# Download from https://rust-lang.org/
cargo install ab-av1
# Set up directories
mkdir C:\Optmiser\tmp, C:\Optmiser\logs
# Run
python C:\Optmiser\optimise_media_v2.py D:\Media\tv tv_movies
```
**On macOS:**
```bash
# Clone repository
git clone https://gitea.theflagroup.com/bnair/VMAFOptimiser.git /opt/Optmiser
---
# Install dependencies
brew install ffmpeg
## Running on Multiple Machines
# Install Rust and ab-av1
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
cargo install ab-av1
### How Lock Files Work
# Set up directories
mkdir -p /opt/Optmiser/tmp /opt/Optmiser/logs /opt/Optmiser/.lock
Each video file has a corresponding lock file:
```
## Running the Optimiser
### Choose Your Script
**Linux / WSL / macOS:**
```bash
# Interactive mode (shows prompts)
./run_optimisation.sh
# Direct execution with parameters
./run_optimisation.sh --directory /mnt/Media/Movies --vmaf 95 --preset 6 --workers 1
# For 50% CPU mode on server
./run_optimisation.sh --directory /mnt/Media/Movies --vmaf 95 --workers 1 --cpu-limit 50
/opt/Optmiser/.lock/{video_filename}.lock
```
**Windows:**
```powershell
# Interactive mode (shows prompts)
.\run_optimisation.ps1
# Direct execution with parameters
.\run_optimisation.ps1 -directory "D:\Movies" --vmaf 95 --preset 6 --workers 1
# For hardware acceleration (AMD GPU)
.\run_optimisation.ps1 -directory "D:\Movies" --vmaf 95 --hwaccel auto
```
## Script Parameters
All wrapper scripts (`run_optimisation.sh` on Linux, `run_optimisation.ps1` on Windows) accept these parameters:
| Parameter | Description | Default |
|------------|-------------|---------|
| `--directory <path>` | Root directory to scan | Current directory |
| `--vmaf <score>` | Target VMAF score | 95.0 |
| `--preset <value>` | SVT-AV1 Preset (4=best, 6=balanced, 8=fast) | 6 |
| `--workers <count>` | Concurrent files to process | 1 |
| `--samples <count>` | Samples for CRF search | 4 |
| `--encoder <name>` | Video encoder: svt-av1, av1_amf, av1_nvenc, av1_qsv | svt-av1 |
| `--hwaccel <value>` | Hardware decode acceleration (auto, d3d11va, vaapi) | none |
| `--use-hardware-worker` | Use 1 HW encoder worker + rest CPU workers | false |
| `--plex-url <url>` | Plex server URL for library refresh | none |
| `--plex-token <token>` | Plex auth token | none |
| `--log-dir <path>` | Log directory | /opt/Optmiser/logs |
## Multi-Machine Setup
### How Lock Files Prevent Conflicts
Each video file has a lock file: `/opt/Optmiser/.lock/{video_filename}`
**Process:**
1. Machine A: Checks for lock → Not found, creates lock
2. Machine A: Starts encoding
3. Machine B: Checks for lock → Found, skips file
4. Machine A: Finishes, removes lock
5. Machine B: Can now process that file
1. Machine A checks for lock → None found, creates lock
2. Machine A starts encoding
3. Machine B checks for lock → Found, skips file
4. Machine A finishes, removes lock
5. Machine B can now process that file
**Result:** Different machines automatically process different files simultaneously!
**Result:** Different machines automatically process different files!
### Setup for Multiple Machines
**Machine 1 - Linux Server (Intel i9-12900H):**
**Machine 1 - Remote Server (Intel i9-12900H):**
```bash
cd /opt/Optmiser
git pull origin main
./run_optimisation.sh /mnt/Media/movies --vmaf 95
# Run on /mnt/Media (Linux filesystem)
python3 optimise_media_v2.py /mnt/Media/tv tv_movies
python3 optimise_media_v2.py /mnt/Media/movies tv_movies
# With 50% CPU limit (recommended)
python3 optimise_media_v2.py /mnt/Media/tv tv_movies --cpu-limit 50
```
**Machine 2 - Windows PC (AMD RX 7900 XT):**
**Machine 2 - Local PC (AMD RX 7900 XT, Windows):**
Option A - Native Windows:
```powershell
# Map network drive if needed
# \\Server\media\ or use local storage
cd C:\Optmiser
git pull origin main
.\run_optimisation.ps1 D:\Media\movies --vmaf 95 --hwaccel auto
# Run on local media
python optimise_media_v2.py D:\Media\tv tv_movies
```
**Machine 3 - Another Linux PC:**
Option B - WSL (Recommended):
```bash
# Windows C: drive accessible at /mnt/c/
cd /opt/Optmiser
python optimise_media_v2.py /mnt/c/Media/tv tv_movies
```
**Machine 3 - Another PC (AMD 9800X3D, Linux):**
```bash
cd /opt/Optmiser
git pull origin main
./run_optimisation.sh /home/user/Media/tv --vmaf 95
# Run on local media directory
python optimise_media_v2.py /home/user/Media/tv tv_movies
```
All three can run simultaneously!
All three can run simultaneously - lock files prevent duplicates!
## Hardware Acceleration
---
### Hardware Decoding vs Hardware Encoding
## Hardware Encoding Guide
There are two types of hardware acceleration:
### Detecting Hardware
1. **Hardware Decoding (`--hwaccel`)**: Uses GPU to decode source video faster. Doesn't affect quality.
2. **Hardware Encoding (`--encoder`)**: Uses GPU to encode output video. Much faster but requires quality compensation.
The script automatically checks for:
1. **AMD GPU encoding support** (future feature)
2. **System thread count**
3. **Operating system** (Linux vs Windows)
### Hardware Encoders
### CPU Encoding (Software)
| Encoder | GPU | Platform | Notes |
|---------|-----|----------|-------|
| `svt-av1` | CPU | All | Default. Best quality, slowest. |
| `av1_amf` | AMD | Windows/Linux | RX 7000 series, ~3-10x faster than CPU |
| `av1_nvenc` | NVIDIA | Windows/Linux | RTX 40 series, very fast |
| `av1_qsv` | Intel | Windows/Linux | Arc GPUs, Intel iGPU with AV1 |
### FFmpeg with Hardware Encoder Support
**Windows (pre-built with HW encoders):**
```powershell
# Most Windows FFmpeg builds include hardware encoders
# Verify with:
ffmpeg -encoders 2>&1 | findstr av1
# Should show: av1_amf, av1_nvenc, av1_qsv (depending on your GPU)
# If missing, download from: https://github.com/BtbN/FFmpeg-Builds/releases
# Choose: ffmpeg-master-latest-win64-gpl.zip (includes all encoders)
```
**Linux (may need custom build):**
```bash
# Check available encoders
ffmpeg -encoders 2>&1 | grep av1
# For AMD (av1_amf) - requires AMF SDK
# Install AMD GPU drivers with AMF support
sudo apt install amdgpu-pro
# For NVIDIA (av1_nvenc) - requires NVIDIA drivers
sudo apt install nvidia-driver-535 # or newer
# For Intel (av1_qsv) - requires Intel Media SDK
sudo apt install intel-media-va-driver-non-free
# If your distro's ffmpeg lacks HW encoders, use static build:
wget https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz
```
### Using Hardware Workers (1 GPU + 3 CPU)
For mixed encoding with your RX 7900 XT:
```powershell
# Windows - 4 workers: 1 uses GPU (av1_amf), 3 use CPU (svt-av1)
.\run_optimisation.ps1 `
-Directory "Z:\" `
-Vmaf 94 `
-Workers 4 `
-Encoder av1_amf `
-Hwaccel auto `
-UseHardwareWorker `
-LogDir "C:\Users\bnair\Documents\VMAFOptimiser\logs"
```
**Best for:** Servers, multi-tasking
```bash
# Linux - same concept
./run_optimisation.sh \
--directory /media \
--vmaf 94 \
--workers 4 \
--encoder av1_amf \
--hwaccel auto \
--use-hardware-worker
# Force CPU encoding
python3 optimise_media_v2.py /media --use-cpu
# Limit to 50% of CPU (12 threads on 24-core)
python3 optimise_media_v2.py /media --cpu-limit 50
# Set specific worker count
python3 optimise_media_v2.py /media --workers 8
```
**How it works:**
- First worker to start claims the GPU and uses `av1_amf`
- Remaining 3 workers use `svt-av1` (CPU)
- GPU worker applies +2 VMAF offset automatically to match CPU quality
**When to use:**
- Running Plex/Jellyfin on same machine
- Server environment with multiple users
- Want to leave GPU free for other tasks
### Quality Compensation
**Expected Speed:** 8-15 fps @ 1080p
Hardware encoders produce lower quality at the same settings. The script automatically compensates:
### GPU Encoding (AMD RX 7900 XT)
| Target VMAF | CPU (svt-av1) | GPU (av1_amf) |
|-------------|---------------|---------------|
| 94 | Searches for VMAF 94 | Searches for VMAF 96 |
| 93 | Searches for VMAF 93 | Searches for VMAF 95 |
| 92 | Searches for VMAF 92 | Searches for VMAF 94 |
**Status:** Planned for future versions
**Expected Speedup:** 3-10x vs CPU encoding
This offset (`HW_ENCODER_VMAF_OFFSET = 2.0`) can be adjusted in `optimize_library.py`.
```bash
# Enable hardware encoding (when implemented)
python3 optimise_media_v2.py /media --use-hardware
```
### Automatic Detection
**When to use:**
- Local PC with AMD GPU
- Want faster encoding speeds
- Can compensate quality with slightly lower VMAF target
When `--hwaccel auto` is specified, the wrapper scripts automatically select the best available hardware acceleration:
**Expected Speed:** 150+ fps @ 1080p
| Platform | Auto Selection | Notes |
|-----------|----------------|--------|
| Windows | d3d11va | Direct3D Video Acceleration |
| macOS | videotoolbox | VideoToolbox framework |
| Linux/WSL | vaapi | Video Acceleration via VA-API |
### Quality Compensation for GPU Encoding
### GPU vs iGPU Priority
GPU encoding may need quality adjustments:
- **Discrete GPU takes priority:** If a discrete GPU (like AMD RX 7900 XT) is present, it's selected over integrated GPU
- **For AMD RX 7900 XT:** Hardware encoding provides ~3-10x speedup over CPU
- **Note:** GPU encoding may need slightly lower VMAF targets to match CPU quality
| CPU VMAF | Equivalent GPU VMAF | Why |
|------------|----------------------|-----|
| 94 | 92-93 | GPU has quality limitations |
| 93 | 91-92 | Hardware encoding trade-offs |
| 90 | 88-89 | Significant compression |
**Recommendation:** If using GPU encoding, set `TARGETS = [92.0, 90.0, 88.0]` for similar quality.
---
## Windows/WSL Path Mapping
### Understanding /mnt/c/
In WSL, Windows drives are mapped:
| Windows | WSL Path | Example |
|---------|------------|---------|
| C:\ | /mnt/c/ | /mnt/c/Users/bnair/Downloads |
| D:\ | /mnt/d/ | /mnt/d/Movies/ |
| E:\ | /mnt/e/ | /mnt/e/TV\ |
**To access Windows media from WSL:**
```bash
# List C:\Media\tv
ls /mnt/c/Media/tv
# Process from WSL
python3 /opt/Optmiser/optimise_media_v2.py /mnt/c/Media/tv tv_movies
```
### Network Drives on WSL
```bash
# Map network drive (one-time)
sudo mkdir -p /mnt/server
echo "//192.168.1.100/media | /mnt/server cifs credentials=/path/to/credfile,uid=1000,gid=1000 0 0" | sudo tee -a /etc/fstab
# Access network media
python3 /opt/Optmiser/optimise_media_v2.py /mnt/server/Media/tv tv_movies
```
---
## Customization
### Changing VMAF Targets
Edit `optimize_library.py`:
Edit `optimise_media_v2.py`:
```python
# Line 15
TARGETS = [94.0, 93.0, 92.0, 90.0]
# More aggressive (smaller files, lower quality)
TARGETS = [92.0, 90.0, 88.0]
@@ -269,16 +265,18 @@ TARGETS = [95.0, 94.0, 93.0]
### Changing Savings Threshold
```python
# More aggressive (encode more)
MIN_SAVINGS_PERCENT = 8.0
# Less aggressive (encode fewer)
MIN_SAVINGS_PERCENT = 15.0
# Line 17
MIN_SAVINGS_PERCENT = 12.0 # Current threshold
MIN_SAVINGS_PERCENT = 8.0 # More aggressive (encode more)
MIN_SAVINGS_PERCENT = 15.0 # Less aggressive (encode fewer)
```
### Changing Encoder Preset
```python
# Line 19
PRESET = 6
# Faster encodes (larger files, lower quality)
PRESET = 8
@@ -286,117 +284,120 @@ PRESET = 8
PRESET = 4
```
## Platform-Specific Tips
### Changing Estimate Target
### For Linux Servers (Intel i9-12900H)
```python
# Line 18
TARGET_SAVINGS_FOR_ESTIMATE = 15.0
1. **Use 50% CPU mode** if running other services:
```bash
./run_optimisation.sh --directory /media --vmaf 95 --workers 1 --cpu-limit 50
```
# Target higher savings
TARGET_SAVINGS_FOR_ESTIMATE = 20.0
```
2. **Run during off-peak hours** to minimize user impact
---
3. **Monitor CPU temperature:**
```bash
watch -n 2 'sensors | grep "Package id"'
```
## Monitoring
4. **Use higher preset for faster encodes:**
```bash
./run_optimisation.sh --vmaf 93 --preset 8
```
### For Windows PCs (AMD RX 7900 XT)
1. **Enable hardware acceleration** for massive speedup:
```powershell
.\run_optimisation.ps1 --directory "D:\Movies" --hwaccel auto
```
2. **Test small sample first** to verify settings:
```powershell
.\run_optimisation.ps1 --directory "D:\Media\sample" --thorough --vmaf 95
```
3. **Monitor GPU usage** (Task Manager or third-party tools)
4. **Consider quality compensation** - GPU encoding may need slightly lower VMAF targets to match CPU quality
### For WSL (Ubuntu/Debian)
1. **Access Windows drives** via `/mnt/c/`:
```bash
ls /mnt/c/Media/movies
```
2. **Increase memory limits** if encoding 4K content:
```bash
# Edit ~/.wslconfig
[wsl2]
memory=16GB
```
## Running in Docker (Optional)
### Watch Progress in Real-Time
```bash
# Build image
docker build -t vmaf-optimiser .
# Tail log file with JSON formatting
tail -f /opt/Optmiser/logs/tv_movies.jsonl | jq '.'
# Run with mount
docker run -v /path/to/media:/media vmaf-optimiser /media
# Monitor encoding speed
watch -n 5 'jq -r "[.savings, .duration]" /opt/Optmiser/logs/tv_movies.jsonl | tail -1'
# Check lock files (what's being processed)
ls -la /opt/Optmiser/.lock/
```
### Performance Dashboard
```bash
# Create a simple dashboard
watch -n 10 '
echo "=== VMAF Optimiser Status ==="
echo ""
echo "Recent Encodes:"
tail -3 /opt/Optmiser/logs/tv_movies.jsonl | jq -r "[.file, .savings, .duration] | @tsv"
echo ""
echo "CPU Usage:"
top -bn1 | head -5
'
```
---
## Troubleshooting
### Issue: Scripts not found
**Solution:** Ensure you're in the correct directory with the scripts installed.
### Issue: "ab-av1: command not found"
**Solution:** Install ab-av1 via cargo:
```bash
cargo install ab-av1
# Check path
ls /opt/Optmiser/optimise_media_v2.py
# Check Python version
python3 --version # Should be 3.8+
# Check ab-av1
/opt/Optmiser/bin/ab-av1 --version # Should be 0.10.3+
```
### Issue: Permission denied
```bash
# Make scripts executable
chmod +x /opt/Optmiser/*.py
chmod +x /opt/Optmiser/*.sh
# Fix lock directory permissions
chmod 777 /opt/Optmiser/.lock
```
### Issue: ab-av1 not found
```bash
# Check if in PATH
which ab-av1
# Use full path if not in PATH
/opt/Optmiser/bin/ab-av1 --version
# Add to PATH
export PATH="$PATH:/opt/Optmiser/bin"
```
### Issue: FFmpeg VMAF not available
**Solution:** Recompile FFmpeg with VMAF support or download a pre-built version that includes libvmaf.
```bash
# Check libvmaf
ffmpeg -filters 2>&1 | grep libvmaf
# If not found, recompile FFmpeg with VMAF
# Or download compiled version from johnvansickle.com/ffmpeg
```
### Issue: Out of Memory
**Solution:** Reduce workers or increase swap:
```bash
# Increase swap
# Check available memory
free -h
# Reduce workers
python3 optimise_media_v2.py /media --workers 4
# Increase swap (if needed)
sudo fallocate -l 4G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
# Use more conservative settings
./run_optimisation.sh --workers 1 --vmaf 93
```
### Issue: Multiple machines encoding same file
**Solution:** This is prevented by lock files. If you see duplicates, check `/opt/Optmiser/.lock/` for stale locks.
### Issue: Encoding fails
**Solution:** Check logs:
```bash
cat /opt/Optmiser/logs/failed_encodes.jsonl | jq '.'
```
### Issue: "unexpected argument" error
**Solution:** Use correct flags for your ab-av1 version. The wrapper scripts now validate support at runtime.
---
## Git Workflow
### Initial Setup
```bash
cd /opt/Optmiser
git init
@@ -408,96 +409,79 @@ git push -u origin main
```
### Daily Updates
```bash
cd /opt/Optmiser
git pull origin main
# Run optimisation
./run_optimisation.sh /media tv_movies
python3 optimise_media_v2.py /media tv_movies
# Review changes
git diff
```
### Committing Changes
### Commit Changes
```bash
cd /opt/Optmiser
git status
# Add changed files
git add optimize_library.py run_optimisation.sh run_optimisation.ps1
git add optimise_media_v2.py run_optimisation.sh
# Commit with message
git commit -m "feat: add Windows and Linux wrapper scripts"
git commit -m "feat: add X"
# Push
git push
```
### Viewing Logs
```bash
# Watch logs in real-time
tail -f /opt/Optmiser/logs/tv_movies.jsonl | jq '.'
# Check files logged for review
cat /opt/Optmiser/logs/low_savings_skips.jsonl | jq '.[] | select(.recommendations=="logged_for_review")'
# Statistics
jq -r '.status' /opt/Optmiser/logs/tv_movies.jsonl | sort | uniq -c
# Find what CRF/VMAF combinations are being used most
jq -r '[.vmaf, .crf] | @tsv' /opt/Optmiser/logs/tv_movies.jsonl | sort | uniq -c
```
### View History
```bash
cd /opt/Optmiser
git log --oneline
git log --graph --all
```
---
## FAQ
**Q: Can I run this on multiple machines at once?**
A: Yes! Each machine will process different files due to lock file mechanism.
**Q: Should I use Windows or WSL?**
A: WSL is recommended for Linux compatibility. Use Windows native if you need direct hardware access or performance.
**Q: Will hardware encoding be available?**
A: Planned for future versions. Currently uses CPU encoding (software AV1).
**Q: Will hardware encoding work better than CPU?**
A: For AMD RX 7900 XT, hardware AV1 encoding is ~3-10x faster than CPU. However, GPU encoding may need slightly lower VMAF targets to match CPU quality.
**Q: How do I know if a file is being encoded elsewhere?**
A: Check `/opt/Optmiser/.lock/` - if a lock exists, another machine is processing it.
**Q: What VMAF target should I use?**
A: Start with VMAF 94 or 95. Drop to 92-90 if you need more savings.
**Q: Can I change the VMAF target?**
A: Yes, edit `TARGETS = [94.0, 93.0, 92.0, 90.0]` in `optimise_media_v2.py`.
**Q: How do I know which files are being processed?**
A: Check `.lock/` directory: `ls -la /opt/Optmiser/.lock/`
**Q: What if savings are always below 12%?**
A: The script will log 15% estimates to `low_savings_skips.jsonl`. Review these logs to decide if encoding is worth it.
**Q: Does this work on Windows/WSL?**
A: Yes! See the Windows/WSL section for setup instructions.
**Q: How much CPU does encoding use?**
A: Full CPU (24 threads) by default. Use `--cpu-limit 50` for 50% mode.
**Q: Can I pause/resume?**
A: Pause by stopping the script (Ctrl+C). Resume by running again - it skips processed files.
**Q: What happens if encoding fails?**
A: Error is logged to `failed_encodes.jsonl`. Original file is NOT modified.
A: Error is logged to `failed_encodes.jsonl` with error code. Original file is NOT modified.
**Q: How much CPU does encoding use?**
A: Full CPU by default. Use `--workers 1` for single-threaded, or limit with `--cpu-limit 50` for 50% (12 threads on 24-core).
---
**Q: What are the log files?**
A:
- `tv_movies.jsonl` - Successful TV & Movie encodes
- `content.jsonl` - Successful Content folder encodes
- `low_savings_skips.jsonl` - Files with <12% savings + 15% estimates
- `failed_searches.jsonl` - Files that couldn't hit any VMAF target
- `failed_encodes.jsonl` - Encoding errors
## Support
**Q: How do I see real-time progress?**
A: The script streams all ab-av1 output in real-time, showing ETA, encoding speed, and frame count.
For issues or questions:
- Check AGENTS.md for detailed technical documentation
- Review logs in `/opt/Optmiser/logs/`
- Test with `--dry-run` flag first
---
**Last Updated:** December 31, 2025
**Version:** 2.0 with Windows and Linux Wrapper Scripts
bin/ab-av1.exe
BIN
View File
Binary file not shown.
debug_status.py
-51
View File
@@ -1,51 +0,0 @@
import os
from pathlib import Path
# Paths
Z_LOGS = Path(r"Z:\.vmaf_logs")
C_LOGS = Path(r"C:\Users\bnair\Videos\encodes\logs")
LOCAL_LOGS = Path("logs").resolve()
Z_LOCKS = Path(r"Z:\.vmaf_locks")
C_LOCKS = Path(r"C:\Users\bnair\Videos\encodes\locks")
print("--- DEBUG STATUS ---")
print(f"\n1. Checking Logs on Z: ({Z_LOGS})")
if Z_LOGS.exists():
print(" [OK] Folder exists.")
for f in Z_LOGS.glob("*"):
print(f" - {f.name} ({f.stat().st_size} bytes)")
else:
print(" [MISSING] Folder does not exist.")
print(f"\n2. Checking Logs on C: ({C_LOGS})")
if C_LOGS.exists():
print(" [OK] Folder exists.")
for f in C_LOGS.glob("*"):
print(f" - {f.name} ({f.stat().st_size} bytes)")
else:
print(" [MISSING] Folder does not exist.")
print(f"\n3. Checking Local Logs: ({LOCAL_LOGS})")
if LOCAL_LOGS.exists():
print(" [OK] Folder exists.")
for f in LOCAL_LOGS.glob("*"):
print(f" - {f.name} ({f.stat().st_size} bytes)")
else:
print(" [MISSING] Folder does not exist.")
print(f"\n4. Checking Locks")
z_count = len(list(Z_LOCKS.glob("*"))) if Z_LOCKS.exists() else 0
c_count = len(list(C_LOCKS.glob("*"))) if C_LOCKS.exists() else 0
print(f" Z: Locks: {z_count}")
print(f" C: Locks: {c_count}")
print(f"\n4. Checking Temp Encodes")
temp = Path(r"C:\Users\bnair\Videos\encodes")
if temp.exists():
mkv_files = list(temp.glob("*.mkv"))
print(f" Found {len(mkv_files)} mkv files.")
for f in mkv_files:
print(f" - {f.name} ({f.stat().st_size / 1024**3:.2f} GB)")
else:
print(" [MISSING] Temp folder missing.")
legacy/finalOptimiser.ps1
-207
View File
@@ -1,207 +0,0 @@
# SMART PRE-FLIGHT ENCODER - VERBOSE CONSOLE OUTPUT
# Usage: .\Encode-TVShows.ps1
param(
[string]$TvDir = "Z:\tv",
[string]$ContentDir = "Z:\content",
[int]$MaxJobs = 2,
[int]$Av1Q = 34,
[int]$HevcQ = 28,
[string]$EncoderAV1 = "av1_amf",
[string]$EncoderHEVC = "hevc_amf",
[switch]$SkipAV1 = $true
)
# --- CONFIGURATION ---
$Global:FFMPEG = "ffmpeg"
$Global:FFPROBE = "ffprobe"
$Global:TEMP_DIR = "C:\Users\bnair\Videos\encodes"
$Global:LockDir = "C:\Users\bnair\Videos\encodes\locks"
$LogFileTV = "C:\Users\bnair\Videos\encodes\encoding-log-tv.csv"
$LogFileContent = "C:\Users\bnair\Videos\encodes\encoding-log-content.csv"
if (-not (Test-Path $Global:TEMP_DIR)) { New-Item -ItemType Directory -Path $Global:TEMP_DIR -Force | Out-Null }
if (-not (Test-Path $Global:LockDir)) { New-Item -ItemType Directory -Path $Global:LockDir -Force | Out-Null }
function Init-LogFile {
param([string]$Path)
if (-not (Test-Path $Path)) { "Timestamp,File,InputSize,OutputSize,CodecUsed,Status,Savings" | Out-File -FilePath $Path -Encoding UTF8 }
}
Init-LogFile $LogFileTV
Init-LogFile $LogFileContent
function Test-Tools {
Write-Host "Setup: Checking required tools..." -ForegroundColor Cyan
if (-not (Get-Command $Global:FFMPEG -ErrorAction SilentlyContinue)) { Write-Host "ERROR: ffmpeg not found!" -ForegroundColor Red; exit 1 }
Write-Host "Setup: Tools found." -ForegroundColor Green
}
$SharedFunctions = {
function Get-LockId {
param([string]$FilePath)
try {
$pathBytes = [System.Text.Encoding]::UTF8.GetBytes($FilePath)
$hash = [System.Security.Cryptography.SHA256]::Create().ComputeHash($pathBytes)
return [BitConverter]::ToString($hash).Replace("-", "").Substring(0, 16)
} catch { return "unknown_lock" }
}
function Run-FFmpeg {
param($In, $Out, $Enc, $Q, $Seek=$null, $Duration=$null)
$argsList = "-hide_banner -loglevel error -y"
if ($Seek) { $argsList += " -ss $Seek" }
$argsList += " -i `"$In`""
if ($Duration) { $argsList += " -t $Duration" }
$argsList += " -c:v $Enc -usage transcoding -quality quality -rc cqp -qp_i $Q -qp_p $Q -qp_b $Q -c:a copy `"$Out`""
return Start-Process -FilePath "ffmpeg" -ArgumentList $argsList -NoNewWindow -Wait -PassThru
}
function Process-VideoFile {
param($InputFile, $CurrentLogFile, $LockDir, $TempDir, $Av1Q, $HevcQ, $EncAV1, $EncHEVC, $SkipAV1)
$pidStr = $PID.ToString()
$lockId = Get-LockId -FilePath $InputFile
$lockFile = Join-Path $LockDir "$lockId.lock"
try {
if (Test-Path $lockFile) { return }
$pidStr | Out-File -FilePath $lockFile -Force
$fileName = Split-Path $InputFile -Leaf
Write-Host "[$pidStr] Found: $fileName" -ForegroundColor White
# Skip Logic
$currentCodec = (& "ffprobe" -v error -select_streams v:0 -show_entries stream=codec_name -of csv=p=0 "$InputFile" 2>&1)
if ($SkipAV1 -and ($currentCodec -match "av1" -or $currentCodec -match "hevc")) {
Write-Host "[$pidStr] SKIP: Already optimized ($currentCodec)" -ForegroundColor DarkGray
return
}
$inputSize = (Get-Item $InputFile).Length
# --- PHASE 1: PRE-FLIGHT SAMPLE ---
Write-Host "[$pidStr] Testing: Generating 60s sample..." -ForegroundColor Yellow
$tempSample = Join-Path $TempDir "$fileName.sample.mkv"
$procSample = Run-FFmpeg $InputFile $tempSample $EncAV1 $Av1Q "00:05:00" "60"
$doFullAV1 = $true
if ($procSample.ExitCode -eq 0 -and (Test-Path $tempSample)) {
$sampleSize = (Get-Item $tempSample).Length
# Threshold: 150MB for 60s is ~20Mbps (Likely bloat)
if ($sampleSize -gt 150MB) {
Write-Host "[$pidStr] Test Result: FAIL. Sample was $([math]::Round($sampleSize/1MB))MB. Too big for AV1." -ForegroundColor Red
$doFullAV1 = $false
} else {
Write-Host "[$pidStr] Test Result: PASS. Sample was $([math]::Round($sampleSize/1MB))MB. Proceeding with AV1." -ForegroundColor Green
}
Remove-Item $tempSample -Force
}
$finalStatus = "Failed"
$finalCodec = "None"
$finalSize = 0
$finalSavings = 0.00
# --- PHASE 2: FULL AV1 ---
if ($doFullAV1) {
Write-Host "[$pidStr] Action: Starting Full AV1 Encode..." -ForegroundColor Cyan
$tempAV1 = Join-Path $TempDir "$fileName.av1.mkv"
$procAV1 = Run-FFmpeg $InputFile $tempAV1 $EncAV1 $Av1Q
if ($procAV1.ExitCode -eq 0 -and (Test-Path $tempAV1)) {
$sizeAV1 = (Get-Item $tempAV1).Length
if ($sizeAV1 -lt $inputSize) {
$finalSavings = [math]::Round((1 - ($sizeAV1 / $inputSize)) * 100, 2)
Write-Host "[$pidStr] AV1 ACCEPTED: Saved ${finalSavings}%" -ForegroundColor Green
$finalOut = $InputFile -replace '\.mkv$', '_av1.mkv' -replace '\.mp4$', '_av1.mp4'
Move-Item $tempAV1 -Destination $finalOut -Force
Remove-Item $InputFile -Force
Rename-Item $finalOut -NewName $fileName -Force
"$(Get-Date -Format 'yyyy-MM-dd HH:mm:ss'),`"$InputFile`",$inputSize,$sizeAV1,AV1,`"Replaced (AV1)`",$finalSavings%" | Out-File -FilePath $CurrentLogFile -Append -Encoding UTF8
return
} else {
Write-Host "[$pidStr] AV1 REJECTED: Larger than source ($([math]::Round($sizeAV1/1GB,2)) GB). Deleting..." -ForegroundColor Red
Remove-Item $tempAV1 -Force
}
}
}
# --- PHASE 3: HEVC FALLBACK ---
Write-Host "[$pidStr] Action: Trying HEVC Fallback..." -ForegroundColor Cyan
$tempHEVC = Join-Path $TempDir "$fileName.hevc.mkv"
$procHEVC = Run-FFmpeg $InputFile $tempHEVC $EncHEVC $HevcQ
if ($procHEVC.ExitCode -eq 0 -and (Test-Path $tempHEVC)) {
$sizeHEVC = (Get-Item $tempHEVC).Length
if ($sizeHEVC -lt $inputSize) {
$finalSavings = [math]::Round((1 - ($sizeHEVC / $inputSize)) * 100, 2)
Write-Host "[$pidStr] HEVC ACCEPTED: Saved ${finalSavings}%" -ForegroundColor Green
$finalOut = $InputFile -replace '\.mkv$', '_hevc.mkv' -replace '\.mp4$', '_hevc.mp4'
Move-Item $tempHEVC -Destination $finalOut -Force
Remove-Item $InputFile -Force
Rename-Item $finalOut -NewName $fileName -Force
$finalStatus = "Replaced (HEVC)"
$finalCodec = "HEVC"
$finalSize = $sizeHEVC
} else {
Write-Host "[$pidStr] HEVC REJECTED: Also larger. Keeping original." -ForegroundColor Red
Remove-Item $tempHEVC -Force
$finalStatus = "Rejected (Both Larger)"
$finalSize = $sizeHEVC
$finalCodec = "HEVC"
}
}
"$(Get-Date -Format 'yyyy-MM-dd HH:mm:ss'),`"$InputFile`",$inputSize,$finalSize,$finalCodec,`"$finalStatus`",$finalSavings%" | Out-File -FilePath $CurrentLogFile -Append -Encoding UTF8
} finally {
Remove-Item $lockFile -Force -ErrorAction SilentlyContinue
}
}
}
function Process-Directory {
param($TargetDirectory, $TargetLogFile, $PhaseName)
if (-not (Test-Path $TargetDirectory)) { return }
Write-Host "`n=== PHASE: $PhaseName ===" -ForegroundColor Magenta
$files = Get-ChildItem -Path $TargetDirectory -Include *.mkv, *.mp4 -Recurse -File -ErrorAction SilentlyContinue
$videoFiles = $files | Where-Object { $_.Name -notmatch "_av1" -and $_.Name -notmatch "_hevc" }
Write-Host "Found $($videoFiles.Count) files." -ForegroundColor Cyan
$processed = 0
while ($processed -lt $videoFiles.Count) {
$batchSize = [math]::Min($MaxJobs, ($videoFiles.Count - $processed))
$currentBatch = $videoFiles[$processed..($processed + $batchSize - 1)]
$jobs = @()
foreach ($file in $currentBatch) {
$jobs += Start-Job -InitializationScript $SharedFunctions -ScriptBlock {
param($f, $log, $lock, $temp, $av1q, $hevcq, $e1, $e2, $skip)
Process-VideoFile $f $log $lock $temp $av1q $hevcq $e1 $e2 $skip
} -ArgumentList $file.FullName, $TargetLogFile, $Global:LockDir, $Global:TEMP_DIR, $Av1Q, $HevcQ, $EncoderAV1, $EncoderHEVC, $SkipAV1
}
while (($jobs | Where-Object { $_.State -eq 'Running' }).Count -gt 0) {
$jobs | Receive-Job
Start-Sleep -Seconds 2
}
$jobs | Receive-Job
$jobs | Remove-Job -Force
$processed += $batchSize
Write-Host "Progress Phase ${PhaseName}: $processed / $($videoFiles.Count)" -ForegroundColor Yellow
}
}
Test-Tools
Process-Directory $TvDir $LogFileTV "TV-Shows"
Process-Directory $ContentDir $LogFileContent "Content"
Write-Host "`nDone." -ForegroundColor Green
legacy/optimize_library.py
-931
View File
@@ -1,931 +0,0 @@
import os
import sys
import subprocess
import argparse
import json
import shutil
import platform
import time
import signal
from pathlib import Path
from datetime import datetime
from concurrent.futures import ThreadPoolExecutor, as_completed, ProcessPoolExecutor
from threading import Lock, get_ident
# --- Configuration ---
DEFAULT_VMAF = 95.0
DEFAULT_PRESET = 6
DEFAULT_WORKERS = 1
DEFAULT_SAMPLES = 4
EXTENSIONS = {".mkv", ".mp4", ".mov", ".avi", ".ts"}
TARGETS = [94.0, 93.0, 92.0, 90.0]
MIN_SAVINGS_PERCENT = 12.0
TARGET_SAVINGS_FOR_ESTIMATE = 15.0
# Hardware encoder quality compensation (GPU needs higher VMAF target to match CPU quality)
HW_ENCODER_VMAF_OFFSET = 2.0
HW_ENCODERS = {"av1_amf", "av1_nvenc", "av1_qsv", "av1_vaapi"}
# Global state for resume capability
_processed_files = set()
_lock = Lock()
_shutdown_requested = False
_AB_AV1_HELP_CACHE = {}
_hw_worker_id = None # Thread ID of the designated hardware worker
def claim_hardware_worker():
"""Attempt to claim hardware worker status for this thread. Returns True if claimed."""
global _hw_worker_id
with _lock:
thread_id = get_ident()
if _hw_worker_id is None:
_hw_worker_id = thread_id
return True
return _hw_worker_id == thread_id
def signal_handler(signum, frame):
"""Handle graceful shutdown"""
global _shutdown_requested
_shutdown_requested = True
print("\n\n⚠️ Shutdown requested. Finishing current tasks...")
signal.signal(signal.SIGINT, signal_handler)
signal.signal(signal.SIGTERM, signal_handler)
def check_dependencies():
missing = []
for tool in ["ffmpeg", "ffprobe", "ab-av1"]:
if not shutil.which(tool):
missing.append(tool)
if missing:
print(f"Error: Missing required tools: {', '.join(missing)}")
print(
"Please install FFmpeg and 'ab-av1' (via cargo install ab-av1) before running."
)
sys.exit(1)
def is_wsl():
if os.environ.get("WSL_DISTRO_NAME"):
return True
try:
with open("/proc/sys/kernel/osrelease", "r", encoding="utf-8") as f:
return "microsoft" in f.read().lower()
except FileNotFoundError:
return False
def platform_label():
system = platform.system()
if system == "Linux" and is_wsl():
return "Linux (WSL)"
return system
def _ab_av1_help(subcommand):
cached = _AB_AV1_HELP_CACHE.get(subcommand)
if cached is not None:
return cached
try:
result = subprocess.run(
["ab-av1", subcommand, "--help"],
capture_output=True,
text=True,
check=False,
)
help_text = (result.stdout or "") + "\n" + (result.stderr or "")
except Exception:
help_text = ""
_AB_AV1_HELP_CACHE[subcommand] = help_text
return help_text
def ab_av1_supports(subcommand, flag):
return flag in _ab_av1_help(subcommand)
def normalize_hwaccel(value):
if value is None:
return None
v = value.strip()
if not v:
return None
v_lower = v.lower()
if v_lower in {"none", "off", "false", "0"}:
return None
if v_lower != "auto":
return v
system = platform.system()
if system == "Windows":
return "d3d11va"
if system == "Darwin":
return "videotoolbox"
return "vaapi"
def get_probe_data(filepath):
"""Get comprehensive video data using ffprobe"""
try:
cmd = [
"ffprobe",
"-v",
"quiet",
"-print_format",
"json",
"-show_streams",
"-show_format",
str(filepath),
]
# FIX: Added encoding="utf-8" and errors="ignore"
res = subprocess.run(
cmd,
capture_output=True,
text=True,
check=True,
encoding="utf-8",
errors="ignore"
)
return json.loads(res.stdout)
except Exception as e:
print(f"Error probing {filepath}: {e}")
return None
def get_video_stats(data):
"""Extract video statistics from ffprobe output"""
if not data or "streams" not in data or "format" not in data:
return None
v_stream = next((s for s in data["streams"] if s["codec_type"] == "video"), None)
if not v_stream:
return None
size = int(data["format"].get("size", 0))
duration = float(data["format"].get("duration", 0))
# Calculate bitrate from file size and duration (more reliable than ffprobe's bitrate)
if size > 0 and duration > 0:
bitrate = int((size * 8) / duration / 1000)
else:
bitrate = int(data["format"].get("bitrate", 0)) // 1000
return {
"codec": v_stream.get("codec_name"),
"width": v_stream.get("width"),
"height": v_stream.get("height"),
"bitrate": bitrate,
"size": size,
"duration": duration,
}
def log_result(log_dir, log_name, data):
"""Log result to JSONL file"""
os.makedirs(log_dir, exist_ok=True)
log_file = Path(log_dir) / f"{log_name}.jsonl"
data["timestamp"] = datetime.now().isoformat()
with _lock:
with open(log_file, "a") as f:
f.write(json.dumps(data) + "\n")
def run_command_streaming(cmd, description=""):
"""Run command and stream output in real-time"""
print(f" [Running {description}]")
process = subprocess.Popen(
cmd,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
universal_newlines=True,
bufsize=1,
)
if process.stdout:
for line in process.stdout:
if _shutdown_requested:
process.terminate()
break
print(f" {line.rstrip()}")
process.wait()
return process.returncode
def run_crf_search(
filepath,
target_vmaf,
preset,
temp_dir,
encoder="svt-av1",
use_hw=False,
hwaccel=None,
):
"""Run CRF search for a specific VMAF target"""
is_hw_encoder = encoder in HW_ENCODERS
# Apply VMAF offset for hardware encoders to match software quality
effective_vmaf = (
target_vmaf + HW_ENCODER_VMAF_OFFSET if is_hw_encoder else target_vmaf
)
# --- Build the Command (Used for both running and parsing) ---
cmd = [
"ab-av1",
"crf-search",
"-i",
str(filepath),
"--min-vmaf",
str(effective_vmaf),
"--preset",
str(preset),
"--max-encoded-percent",
"100",
"--temp-dir",
temp_dir,
"--samples",
"4",
]
# FORCE use of the specified encoder (Fixes the CPU default crash)
if ab_av1_supports("crf-search", "--encoder"):
cmd.extend(["--encoder", encoder])
# Hardware decode acceleration
if use_hw and hwaccel:
if ab_av1_supports("crf-search", "--enc-input"):
cmd.extend(["--enc-input", f"hwaccel={hwaccel}"])
if hwaccel == "vaapi":
cmd.extend(["--enc-input", "hwaccel_output_format=vaapi"])
vmaf_label = f"VMAF {effective_vmaf}" if is_hw_encoder else f"VMAF {target_vmaf}"
print(f" - Searching for CRF to hit {vmaf_label}...")
# 1. First Run: Stream output to console
returncode = run_command_streaming(cmd, f"crf-search {vmaf_label}")
if returncode == 0:
# 2. Second Run: Capture output to parse the CRF
# FIX: Reuse 'cmd' and add encoding parameters to prevent charmap crashes
res = subprocess.run(
cmd,
capture_output=True,
text=True,
encoding="utf-8",
errors="ignore"
)
lines = res.stdout.strip().split("\n")
for line in reversed(lines):
if "crf" in line.lower():
try:
parts = line.split()
crf_val = float(parts[1])
percent = 100.0
for p in parts:
if "%" in p:
percent = float(p.strip("()%"))
break
return {
"crf": crf_val,
"predicted_percent": percent,
"vmaf": target_vmaf,
}
except Exception as e:
print(f" ! Failed to parse crf-search output: {e}")
return None
def find_target_savings_params(
filepath,
start_vmaf,
preset,
temp_dir,
encoder="svt-av1",
use_hw=False,
hwaccel=None,
):
"""Find VMAF target that achieves minimum savings"""
print(f"\n --- Finding VMAF for {TARGET_SAVINGS_FOR_ESTIMATE}% savings ---")
test_targets = [t for t in TARGETS if t <= start_vmaf]
for i, target in enumerate(test_targets):
if _shutdown_requested:
return None
print(
f" Testing VMAF {target} for {TARGET_SAVINGS_FOR_ESTIMATE}% target... (test {i + 1}/{len(test_targets)})"
)
result = run_crf_search(
filepath, target, preset, temp_dir, encoder, use_hw, hwaccel
)
if result:
predicted_savings = 100.0 - result["predicted_percent"]
quality_drop = start_vmaf - target
print(
f" ✓ VMAF {target}: CRF {result['crf']}, Savings {predicted_savings:.1f}%, Drop -{quality_drop:.0f}"
)
if predicted_savings >= TARGET_SAVINGS_FOR_ESTIMATE:
print(f"\n ✅ FOUND {TARGET_SAVINGS_FOR_ESTIMATE}%+ SAVINGS:")
print(f" Target VMAF: {target} (quality drop: -{quality_drop:.0f})")
print(f" CRF: {result['crf']}")
print(f" Predicted savings: {predicted_savings:.1f}%")
return {
"target_vmaf": target,
"crf": result["crf"],
"savings": predicted_savings,
"quality_drop": quality_drop,
"found": True,
}
else:
print(f" ✗ Could not achieve VMAF {target}")
print(f"\n 📝 COULD NOT ACHIEVE {TARGET_SAVINGS_FOR_ESTIMATE}% SAVINGS")
print(f" Tried VMAF targets: {test_targets}")
return None
def run_encode(
filepath, output_path, crf, preset, encoder="svt-av1", use_hw=False, hwaccel=None
):
"""Run full encoding with real-time output streaming"""
cmd = [
"ab-av1",
"encode",
"--input",
str(filepath),
"--output",
str(output_path),
"--crf",
str(crf),
"--preset",
str(preset),
"--acodec",
"copy",
]
if encoder != "svt-av1":
if ab_av1_supports("encode", "--encoder"):
cmd.extend(["--encoder", encoder])
if use_hw and hwaccel:
if ab_av1_supports("encode", "--enc-input"):
cmd.extend(["--enc-input", f"hwaccel={hwaccel}"])
if hwaccel == "vaapi":
cmd.extend(["--enc-input", "hwaccel_output_format=vaapi"])
return run_command_streaming(cmd, f"encoding (CRF {crf})")
def provide_recommendations(
stats_before, hit_vmaf, predicted_savings, target_result=None
):
"""Provide recommendations based on analysis results"""
print(f"\n --- Recommendations for {stats_before['codec']} content ---")
if target_result and target_result["found"]:
print(f" 📊 TO HIT {TARGET_SAVINGS_FOR_ESTIMATE}% SAVINGS:")
print(
f" → Target VMAF: {target_result['target_vmaf']} (drop: -{target_result['quality_drop']:.0f})"
)
print(f" → CRF: {target_result['crf']}")
print(f" → Predicted: {target_result['savings']:.1f}% savings")
print(f" → Trade-off: Quality reduction for space savings")
print()
if stats_before["bitrate"] < 2000:
print(f" → Source bitrate is low ({stats_before['bitrate']}k)")
print(f" → AV1 gains minimal on highly compressed sources")
print(f" → Recommendation: SKIP - Source already optimized")
return
if stats_before["codec"] in ["hevc", "h265", "vp9"]:
print(f" → Source already uses modern codec ({stats_before['codec']})")
print(f" → AV1 gains minimal on already-compressed content")
print(f" → Recommendation: SKIP - Codec already efficient")
return
if target_result and not target_result["found"]:
print(
f" → Could not achieve {TARGET_SAVINGS_FOR_ESTIMATE}% even at lowest VMAF"
)
print(f" → Content may not compress well with AV1")
print(f" → Recommendation: SKIP - Review manually")
def refresh_plex(plex_url, plex_token):
"""Refresh Plex library after encoding completion"""
if not plex_url or not plex_token:
return
try:
print("\n📡 Refreshing Plex library...")
cmd = [
"curl",
"-X",
"GET",
f"{plex_url}/library/sections/1/refresh",
"-H",
f"X-Plex-Token: {plex_token}",
]
subprocess.run(cmd, capture_output=True, check=False)
print(" ✓ Plex refresh triggered")
except Exception as e:
print(f" ⚠️ Failed to refresh Plex: {e}")
def process_file(
filepath,
log_dir,
log_name,
preset,
hw_encoder="svt-av1",
use_hw_mode=False,
hwaccel=None,
):
"""Process a single video file with intelligent VMAF targeting"""
global _shutdown_requested
# --- LOGIC FIX START ---
# If "use_hw_mode" (Hybrid) is FALSE, use the requested encoder for EVERYONE.
if not use_hw_mode:
encoder = hw_encoder
use_hw = (encoder in HW_ENCODERS)
else:
# Hybrid Mode: Only 1 worker gets HW, rest get CPU
use_hw = False
if hwaccel and hw_encoder in HW_ENCODERS:
use_hw = claim_hardware_worker()
encoder = hw_encoder if use_hw else "svt-av1"
# --- LOGIC FIX END ---
filepath = Path(filepath)
lock_file = Path(log_dir).parent / ".lock" / f"{filepath.name}.lock"
# Check lock file (multi-machine coordination)
if lock_file.exists():
print(f"Skipping (Locked): {filepath.name}")
return True
# Create lock
lock_file.parent.mkdir(parents=True, exist_ok=True)
lock_file.touch()
try:
probe_data = get_probe_data(filepath)
if not probe_data:
print(f"Skipping (Invalid probe data): {filepath.name}")
return True
stats_before = get_video_stats(probe_data)
if not stats_before or stats_before["codec"] == "av1":
print(f"Skipping (Already AV1 or invalid): {filepath.name}")
return True
# Mark as processed
file_key = str(filepath)
with _lock:
if file_key in _processed_files:
return True
_processed_files.add(file_key)
print(f"\n--- Processing: {filepath.name} ---")
# Added 'errors="ignore"' to print to avoid Unicode crashes on console
try:
print(
f" Source: {stats_before['codec']} @ {stats_before['bitrate']}k, {stats_before['size'] / (1024**3):.2f} GB"
)
except UnicodeEncodeError:
print(f" Source: {stats_before['codec']} @ {stats_before['bitrate']}k")
if _shutdown_requested:
return False
temp_dir = Path(log_dir).parent / "tmp"
temp_dir.mkdir(exist_ok=True)
# Step 1: Try VMAF 94
print(f"\n [Step 1] Testing VMAF 94...")
search_result_94 = run_crf_search(
filepath, 94.0, preset, str(temp_dir), encoder, use_hw, hwaccel
)
if not search_result_94:
print(f" !! Could not hit VMAF 94")
search_result_94 = run_crf_search(
filepath, 93.0, preset, str(temp_dir), encoder, use_hw, hwaccel
)
if not search_result_94:
search_result_94 = run_crf_search(
filepath, 92.0, preset, str(temp_dir), encoder, use_hw, hwaccel
)
if not search_result_94:
search_result_94 = run_crf_search(
filepath, 90.0, preset, str(temp_dir), encoder, use_hw, hwaccel
)
if not search_result_94:
print(f" !! Failed all VMAF targets ({TARGETS}) for {filepath.name}")
log_result(
log_dir,
"failed_searches",
{
"file": str(filepath),
"status": "all_targets_failed",
"targets": TARGETS,
},
)
provide_recommendations(stats_before, None, 0)
return False
crf_94 = search_result_94["crf"]
predicted_savings_94 = 100.0 - search_result_94["predicted_percent"]
if predicted_savings_94 >= MIN_SAVINGS_PERCENT:
print(
f"\n ✅ VMAF 94 gives {predicted_savings_94:.1f}% savings (≥{MIN_SAVINGS_PERCENT}%)"
)
print(f" → Proceeding with VMAF 94, CRF {crf_94}")
encode_params = {
"crf": crf_94,
"vmaf": 94.0,
"predicted_percent": search_result_94["predicted_percent"],
}
else:
print(
f"\n ⚠️ VMAF 94 gives {predicted_savings_94:.1f}% savings (<{MIN_SAVINGS_PERCENT}%)"
)
search_result_93 = run_crf_search(
filepath, 93.0, preset, str(temp_dir), encoder, use_hw, hwaccel
)
if search_result_93:
predicted_savings_93 = 100.0 - search_result_93["predicted_percent"]
if predicted_savings_93 >= MIN_SAVINGS_PERCENT:
print(
f" ✅ VMAF 93 gives {predicted_savings_93:.1f}% savings (≥{MIN_SAVINGS_PERCENT}%)"
)
print(
f" → Proceeding with VMAF 93, CRF {search_result_93['crf']}"
)
encode_params = {
"crf": search_result_93["crf"],
"vmaf": 93.0,
"predicted_percent": search_result_93["predicted_percent"],
}
else:
print(
f" ⚠️ VMAF 93 gives {predicted_savings_93:.1f}% savings (also <{MIN_SAVINGS_PERCENT}%)"
)
print(
f" → Finding VMAF for {TARGET_SAVINGS_FOR_ESTIMATE}% savings..."
)
target_result = find_target_savings_params(
filepath, 93.0, preset, str(temp_dir), encoder, use_hw, hwaccel
)
provide_recommendations(
stats_before, 93.0, predicted_savings_93, target_result
)
log_result(
log_dir,
"low_savings_skips",
{
"file": str(filepath),
"vmaf_94": 94.0,
"savings_94": predicted_savings_94,
"vmaf_93": 93.0,
"savings_93": predicted_savings_93,
"target_for_15_percent": target_result,
"recommendations": "logged_for_review",
},
)
return True
else:
print(f" !! Could not achieve VMAF 93")
log_result(
log_dir,
"failed_searches",
{"file": str(filepath), "status": "vmaf_93_failed"},
)
return False
temp_output = temp_dir / f"{filepath.stem}.av1_temp.mkv"
if temp_output.exists():
temp_output.unlink()
start_time = time.time()
res = run_encode(
filepath,
temp_output,
encode_params["crf"],
preset,
encoder,
use_hw,
hwaccel,
)
if res != 0:
print(f"\n !! Encode failed with code {res}")
if temp_output.exists():
temp_output.unlink()
log_result(
log_dir,
"failed_encodes",
{"file": str(filepath), "status": "encode_failed", "returncode": res},
)
return False
encode_duration = time.time() - start_time
print(f" ✓ Encode completed in {encode_duration:.1f}s")
probe_after = get_probe_data(temp_output)
stats_after = get_video_stats(probe_after)
if not stats_after:
print(f" !! Failed to probe encoded file")
if temp_output.exists():
temp_output.unlink()
return False
actual_savings = (1 - (stats_after["size"] / stats_before["size"])) * 100
print(f"\n Results:")
print(
f" - Before: {stats_before['size'] / (1024**3):.2f} GB @ {stats_before['bitrate']}k"
)
print(
f" - After: {stats_after['size'] / (1024**3):.2f} GB @ {stats_after['bitrate']}k"
)
print(f" - Savings: {actual_savings:.2f}%")
final_path = filepath
if filepath.suffix.lower() == ".mp4":
final_path = filepath.with_suffix(".mkv")
if final_path.exists():
final_path.unlink()
shutil.move(str(filepath), str(final_path))
shutil.move(str(temp_output), str(final_path))
print(f" ✓ Successfully optimized: {final_path.name}")
log_result(
log_dir,
log_name,
{
"file": str(final_path),
"status": "success",
"vmaf": encode_params["vmaf"],
"crf": encode_params["crf"],
"before": stats_before,
"after": stats_after,
"duration": encode_duration,
"savings": actual_savings,
},
)
return True
finally:
# Remove lock file
if lock_file.exists():
lock_file.unlink()
def scan_library(root, exclude_dirs=None):
"""Scan library for video files, excluding certain directories"""
exclude_dirs = exclude_dirs or []
files = []
for dirpath, dirnames, filenames in os.walk(root):
# Skip excluded directories
dirnames[:] = [d for d in dirnames if d not in exclude_dirs]
for filename in filenames:
if Path(filename).suffix.lower() not in EXTENSIONS:
continue
full_path = Path(dirpath) / filename
if "_av1" in full_path.stem:
continue
files.append(full_path)
return files
def main():
parser = argparse.ArgumentParser(
description="Optimize video library to AV1 using VMAF targeting."
)
parser.add_argument("directory", help="Root directory to scan")
parser.add_argument(
"--vmaf",
type=float,
default=DEFAULT_VMAF,
help=f"Target VMAF score (default: {DEFAULT_VMAF})",
)
parser.add_argument(
"--preset",
type=int,
default=DEFAULT_PRESET,
help=f"SVT-AV1 Preset (default: {DEFAULT_PRESET})",
)
parser.add_argument(
"--workers",
type=int,
default=DEFAULT_WORKERS,
help=f"Concurrent files to process (default: {DEFAULT_WORKERS})",
)
parser.add_argument(
"--samples",
type=int,
default=DEFAULT_SAMPLES,
help=f"Samples to use for CRF search if supported (default: {DEFAULT_SAMPLES})",
)
parser.add_argument(
"--hwaccel",
default=None,
help=(
"Hardware acceleration for decode. "
"Examples: auto, vaapi, d3d11va, videotoolbox. Use 'none' to disable."
),
)
parser.add_argument(
"--encoder",
default="svt-av1",
help=(
"Video encoder to use. Default: svt-av1 (CPU). "
"Hardware encoders: av1_amf (AMD), av1_nvenc (NVIDIA), av1_qsv (Intel)."
),
)
parser.add_argument(
"--use-hardware-worker",
action="store_true",
help="Use 1 hardware encoding worker + rest CPU workers (requires --encoder with HW encoder)",
)
parser.add_argument(
"--plex-url",
default=None,
help="Plex server URL (e.g., http://localhost:32400)",
)
parser.add_argument(
"--plex-token",
default=None,
help="Plex auth token",
)
parser.add_argument(
"--log-dir",
default="/opt/Optmiser/logs",
help="Log directory (default: /opt/Optmiser/logs)",
)
args = parser.parse_args()
if args.workers < 1:
print("Error: --workers must be >= 1")
sys.exit(2)
check_dependencies()
root = Path(args.directory)
if not root.exists():
print(f"Directory not found: {root}")
sys.exit(1)
hwaccel = normalize_hwaccel(args.hwaccel)
print(f"Platform: {platform_label()}")
print(f"Scanning library: {root}")
print(f"VMAF targets: {TARGETS}")
print(f"Minimum savings: {MIN_SAVINGS_PERCENT}%")
print(f"Estimate target: {TARGET_SAVINGS_FOR_ESTIMATE}%")
print(f"Encoder Preset: {args.preset}")
print(f"Workers: {args.workers}")
if hwaccel:
print(f"HWAccel: {hwaccel}")
if args.use_hardware_worker:
print(f"Hardware worker: 1 HW + {args.workers - 1} CPU")
if args.plex_url:
print(f"Plex refresh: Enabled")
print("-" * 60)
# Determine log name based on directory
root_parts = str(root).lower().split("/")
if "content" in root_parts:
log_name = "content"
exclude_dirs = []
else:
log_name = "tv_movies"
exclude_dirs = ["content"]
print(f"Log file: {log_name}.jsonl")
files = scan_library(root, exclude_dirs)
if not files:
print("No media files found.")
return
print(f"Found {len(files)} files to process")
print("-" * 60)
processed_count = 0
success_count = 0
fail_count = 0
# Hardware worker configuration
# HW worker uses specified encoder; CPU workers use svt-av1
hw_encoder = args.encoder if args.encoder in HW_ENCODERS else None
use_hw_primary = args.use_hardware_worker and hw_encoder is not None
if args.workers == 1:
# Single thread - just process files
for file_path in files:
if _shutdown_requested:
break
processed_count += 1
result = process_file(
file_path,
args.log_dir,
log_name,
args.preset,
args.encoder,
use_hw_primary,
hwaccel,
)
if result:
success_count += 1
else:
fail_count += 1
else:
# Multi-threaded processing
with ThreadPoolExecutor(max_workers=args.workers) as executor:
futures = []
for file_path in files:
if _shutdown_requested:
break
# All workers try to claim HW; only the first thread succeeds
# and will use HW for ALL its tasks
future = executor.submit(
process_file,
file_path,
args.log_dir,
log_name,
args.preset,
args.encoder,
use_hw_primary,
hwaccel,
)
futures.append(future)
for future in as_completed(futures):
if _shutdown_requested:
break
processed_count += 1
try:
result = future.result()
if result:
success_count += 1
else:
fail_count += 1
except Exception as e:
print(f" !! Unexpected error: {e}")
import traceback
traceback.print_exc()
fail_count += 1
print("\n" + "=" * 60)
print(f"SUMMARY: {root}")
print(f" Processed: {processed_count} files")
print(f" Success/Skip: {success_count}")
print(f" Failed: {fail_count}")
print("=" * 60)
# Refresh Plex on completion
if success_count > 0:
refresh_plex(args.plex_url, args.plex_token)
if __name__ == "__main__":
main()
legacy/run_optimisation.ps1
-100
View File
@@ -1,100 +0,0 @@
param(
[Parameter(Mandatory=$false)]
[string]$Directory = ".",
[float]$Vmaf = 95.0,
[int]$Preset = 6,
[int]$Workers = 1,
[int]$Samples = 4,
[string]$Hwaccel = "",
[string]$Encoder = "svt-av1",
[switch]$UseHardwareWorker,
[string]$PlexUrl = "",
[string]$PlexToken = "",
[string]$LogDir = "/opt/Optmiser/logs"
)
$ErrorActionPreference = "Stop"
function Write-ColorOutput {
param([string]$Message, [string]$Color = "White")
Write-Host $Message -ForegroundColor $Color
}
function Invoke-OptimizeLibrary {
$scriptPath = Join-Path $PSScriptRoot "optimize_library.py"
if (-not (Test-Path $scriptPath)) {
Write-ColorOutput -Message "ERROR: optimize_library.py not found in current directory" -Color "Red"
exit 1
}
$pythonCmd = Get-Command "python"
if (-not $pythonCmd) {
Write-ColorOutput -Message "ERROR: Python 3 not found. Please install Python 3." -Color "Red"
exit 1
}
$arguments = @(
$scriptPath,
$Directory,
"--vmaf", $Vmaf.ToString("F1"),
"--preset", $Preset.ToString(),
"--workers", $Workers.ToString(),
"--samples", $Samples.ToString(),
"--encoder", $Encoder,
"--log-dir", $LogDir
)
if ($Hwaccel) {
$arguments += "--hwaccel", $Hwaccel
}
if ($UseHardwareWorker) {
$arguments += "--use-hardware-worker"
}
if ($PlexUrl) {
$arguments += "--plex-url", $PlexUrl
}
if ($PlexToken) {
$arguments += "--plex-token", $PlexToken
}
Write-ColorOutput -Message "Running optimize_library.py..." -Color "Cyan"
Write-ColorOutput -Message " Directory: $Directory" -Color "White"
Write-ColorOutput -Message " Target VMAF: $Vmaf" -Color "White"
Write-ColorOutput -Message " Preset: $Preset" -Color "White"
Write-ColorOutput -Message " Workers: $Workers" -Color "White"
Write-ColorOutput -Message " Samples: $Samples" -Color "White"
Write-ColorOutput -Message " Encoder: $Encoder" -Color "White"
if ($Hwaccel) {
Write-ColorOutput -Message " HW Accel: $Hwaccel" -Color "White"
}
if ($UseHardwareWorker) {
Write-ColorOutput -Message " Hardware worker: Enabled (1 HW + $($Workers - 1) CPU)" -Color "White"
}
if ($PlexUrl -and $PlexToken) {
Write-ColorOutput -Message " Plex refresh: Enabled" -Color "White"
}
Write-Host ""
$process = Start-Process -FilePath $pythonCmd.Path -ArgumentList $arguments -NoNewWindow -PassThru
$process.WaitForExit()
$exitCode = $process.ExitCode
if ($exitCode -eq 0) {
Write-ColorOutput -Message "SUCCESS: Library optimization completed" -Color "Green"
} else {
Write-ColorOutput -Message "ERROR: optimize_library.py exited with code $exitCode" -Color "Red"
}
exit $exitCode
}
Write-ColorOutput -Message "========================================" -Color "Cyan"
Write-ColorOutput -Message "VMAF Library Optimiser (Windows)" -Color "Yellow"
Write-ColorOutput -Message "========================================" -Color "Cyan"
Write-Host ""
Invoke-OptimizeLibrary
legacy/run_optimisation.sh
-158
View File
@@ -1,158 +0,0 @@
#!/bin/bash
set -e
COLOR_RED='\033[0;31m'
COLOR_GREEN='\033[0;32m'
COLOR_CYAN='\033[0;36m'
COLOR_YELLOW='\033[1;33m'
COLOR_WHITE='\033[0;37m'
COLOR_RESET='\033[0m'
log_info() {
echo -e "${COLOR_CYAN}$*${COLOR_RESET}"
}
log_error() {
echo -e "${COLOR_RED}ERROR: $*${COLOR_RESET}" >&2
}
log_success() {
echo -e "${COLOR_GREEN}$*${COLOR_RESET}"
}
DIRECTORY="."
VMAF="95.0"
PRESET="6"
WORKERS="1"
SAMPLES="4"
HWACCEL=""
USE_HW_WORKER=""
PLEX_URL=""
PLEX_TOKEN=""
LOG_DIR="/opt/Optmiser/logs"
# Parse command line arguments
while [[ $# -gt 0 ]]; do
case "$1" in
--directory)
DIRECTORY="$2"
shift 2
;;
--vmaf)
VMAF="$2"
shift 2
;;
--preset)
PRESET="$2"
shift 2
;;
--workers)
WORKERS="$2"
shift 2
;;
--samples)
SAMPLES="$2"
shift 2
;;
--hwaccel)
HWACCEL="$2"
shift 2
;;
--use-hardware-worker)
USE_HARDWARE_WORKER="true"
shift
;;
--plex-url)
PLEX_URL="$2"
shift 2
;;
--plex-token)
PLEX_TOKEN="$2"
shift 2
;;
--log-dir)
LOG_DIR="$2"
shift 2
;;
*)
DIRECTORY="$1"
shift
;;
esac
done
# Check if python3 is available
if ! command -v python3 &> /dev/null; then
if ! command -v python &> /dev/null; then
log_error "Python 3 not found. Please install Python 3."
exit 1
else
PYTHON_CMD="python"
fi
else
PYTHON_CMD="python3"
fi
# Check if optimize_library.py exists
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
SCRIPT_PATH="$SCRIPT_DIR/optimize_library.py"
if [[ ! -f "$SCRIPT_PATH" ]]; then
log_error "optimize_library.py not found in: $SCRIPT_DIR"
exit 1
fi
# Build command arguments
ARGS=(
"$PYTHON_CMD" "$SCRIPT_PATH"
"$DIRECTORY"
--vmaf "$VMAF"
--preset "$PRESET"
--workers "$WORKERS"
--samples "$SAMPLES"
--log-dir "$LOG_DIR"
)
if [[ -n "$THOROUGH" ]]; then
ARGS+=(--thorough)
fi
if [[ -n "$HWACCEL" ]]; then
ARGS+=(--hwaccel "$HWACCEL")
fi
# Print configuration
log_info "========================================"
log_info "VMAF Library Optimiser (Linux/Server)"
log_info "========================================"
echo ""
log_info "Directory: $DIRECTORY"
log_info "Target VMAF: $VMAF"
log_info "Preset: $PRESET"
log_info "Workers: $WORKERS"
log_info "Samples: $SAMPLES"
log_info "Encoder: $ENCODER"
if [[ -n "$THOROUGH" ]]; then
log_info "Thorough: Yes"
fi
if [[ -n "$HWACCEL" ]]; then
log_info "HW Accel: $HWACCEL"
fi
echo ""
log_info "Running optimize_library.py..."
echo ""
# Run the optimisation
"${ARGS[@]}"
EXIT_CODE=$?
# Handle exit code
if [ $EXIT_CODE -eq 0 ]; then
log_success "SUCCESS: Library optimisation completed"
else
log_error "optimize_library.py exited with code $EXIT_CODE"
fi
exit $EXIT_CODE
legacy/run_smart_encoder.ps1
-30
View File
@@ -1,30 +0,0 @@
# Run Smart GPU Encoder
# Wrapper to ensure correct environment
$ScriptPath = "$PSScriptRoot\smart_gpu_encoder.py"
$PythonPath = "python" # Or specific path like "C:\Python39\python.exe"
# Directories (Change these to your actual paths)
$TvDir = "Z:\tv"
$ContentDir = "Z:\content"
Write-Host "==========================================" -ForegroundColor Cyan
Write-Host " SMART GPU ENCODER (AMD AMF + VMAF)" -ForegroundColor Cyan
Write-Host "==========================================" -ForegroundColor Cyan
Write-Host "Script: $ScriptPath"
Write-Host "TV Dir: $TvDir"
Write-Host "Content Dir: $ContentDir"
Write-Host ""
# Check if ab-av1 exists in bin
if (-not (Test-Path "$PSScriptRoot\bin\ab-av1.exe")) {
Write-Host "WARNING: ab-av1.exe not found in bin folder!" -ForegroundColor Red
Write-Host "Please download it and place it in $PSScriptRoot\bin\"
exit 1
}
# Run the python script
& $PythonPath $ScriptPath --tv-dir $TvDir --content-dir $ContentDir
Write-Host "`nDone." -ForegroundColor Green
Read-Host "Press Enter to exit..."
optimize_library.py
+341
View File
@@ -0,0 +1,341 @@
import os
import sys
import subprocess
import argparse
import json
import shutil
import platform
from concurrent.futures import ThreadPoolExecutor, as_completed
from pathlib import Path
DEFAULT_VMAF = 95.0
DEFAULT_PRESET = 6
DEFAULT_WORKERS = 1
DEFAULT_SAMPLES = 4
EXTENSIONS = {".mkv", ".mp4", ".mov", ".avi", ".ts"}
_AB_AV1_HELP_CACHE = {}
def check_dependencies():
missing = []
for tool in ["ffmpeg", "ffprobe", "ab-av1"]:
if not shutil.which(tool):
missing.append(tool)
if missing:
print(f"Error: Missing required tools: {', '.join(missing)}")
print(
"Please install FFmpeg and 'ab-av1' (via cargo install ab-av1) before running."
)
sys.exit(1)
def is_wsl():
if os.environ.get("WSL_DISTRO_NAME"):
return True
try:
with open("/proc/sys/kernel/osrelease", "r", encoding="utf-8") as f:
return "microsoft" in f.read().lower()
except FileNotFoundError:
return False
def platform_label():
system = platform.system()
if system == "Linux" and is_wsl():
return "Linux (WSL)"
return system
def _ab_av1_help(subcommand):
cached = _AB_AV1_HELP_CACHE.get(subcommand)
if cached is not None:
return cached
try:
result = subprocess.run(
["ab-av1", subcommand, "--help"],
capture_output=True,
text=True,
check=False,
)
help_text = (result.stdout or "") + "\n" + (result.stderr or "")
except Exception:
help_text = ""
_AB_AV1_HELP_CACHE[subcommand] = help_text
return help_text
def ab_av1_supports(subcommand, flag):
return flag in _ab_av1_help(subcommand)
def normalize_hwaccel(value):
if value is None:
return None
v = value.strip()
if not v:
return None
v_lower = v.lower()
if v_lower in {"none", "off", "false", "0"}:
return None
if v_lower != "auto":
return v
system = platform.system()
if system == "Windows":
return "d3d11va"
if system == "Darwin":
return "videotoolbox"
return "vaapi"
def get_video_info(filepath):
try:
cmd = [
"ffprobe",
"-v",
"quiet",
"-print_format",
"json",
"-show_streams",
"-select_streams",
"v:0",
filepath,
]
result = subprocess.run(cmd, capture_output=True, text=True, check=True)
data = json.loads(result.stdout)
streams = data.get("streams") or []
if not streams:
return None
stream = streams[0]
codec = stream.get("codec_name", "unknown")
color_transfer = stream.get("color_transfer", "unknown")
is_hdr = color_transfer in ["smpte2084", "arib-std-b67"]
return {"codec": codec, "is_hdr": is_hdr}
except Exception as e:
print(f"Error probing {filepath}: {e}")
return None
def build_ab_av1_command(input_path, output_path, args):
cmd = [
"ab-av1",
"auto-encode",
"-i",
str(input_path),
"-o",
str(output_path),
"--min-vmaf",
str(args.vmaf),
"--preset",
str(args.preset),
]
if args.encoder:
if ab_av1_supports("auto-encode", "--encoder"):
cmd.extend(["--encoder", args.encoder])
elif ab_av1_supports("auto-encode", "-e"):
cmd.extend(["-e", args.encoder])
else:
print("Warning: This ab-av1 version does not support --encoder; ignoring.")
if args.samples is not None:
if ab_av1_supports("auto-encode", "--samples"):
cmd.extend(["--samples", str(args.samples)])
elif ab_av1_supports("auto-encode", "--sample-count"):
cmd.extend(["--sample-count", str(args.samples)])
else:
print("Warning: This ab-av1 version does not support --samples; ignoring.")
if args.thorough:
if ab_av1_supports("auto-encode", "--thorough"):
cmd.append("--thorough")
else:
print("Warning: This ab-av1 version does not support --thorough; ignoring.")
hwaccel = normalize_hwaccel(args.hwaccel)
if hwaccel is not None:
if ab_av1_supports("auto-encode", "--enc-input"):
cmd.extend(["--enc-input", f"hwaccel={hwaccel}"])
hwaccel_output_format = args.hwaccel_output_format
if hwaccel_output_format is None and hwaccel == "vaapi":
hwaccel_output_format = "vaapi"
if hwaccel_output_format is not None:
cmd.extend(
["--enc-input", f"hwaccel_output_format={hwaccel_output_format}"]
)
else:
print(
"Warning: This ab-av1 version does not support --enc-input; ignoring --hwaccel."
)
if ab_av1_supports("auto-encode", "--acodec"):
cmd.extend(["--acodec", "copy"])
elif ab_av1_supports("auto-encode", "--ac"):
cmd.extend(["--ac", "copy"])
else:
print(
"Warning: This ab-av1 version does not support --acodec/--ac; leaving audio defaults."
)
return cmd
def process_file(filepath, args):
input_path = Path(filepath)
output_path = input_path.with_stem(input_path.stem + "_av1")
if output_path.exists():
print(f"Skipping (Output exists): {input_path.name}")
return
info = get_video_info(str(input_path))
if not info:
return
if info["codec"] == "av1":
print(f"Skipping (Already AV1): {input_path.name}")
return
print(f"\nProcessing: {input_path.name}")
print(f" Source Codec: {info['codec']}")
print(f" HDR: {info['is_hdr']}")
cmd = build_ab_av1_command(input_path, output_path, args)
try:
subprocess.run(cmd, check=True)
print(f"Success! Encoded: {output_path.name}")
except subprocess.CalledProcessError:
print(f"Failed to encode: {input_path.name}")
if output_path.exists():
os.remove(output_path)
def scan_library(root):
files = []
for dirpath, _, filenames in os.walk(root):
for filename in filenames:
if Path(filename).suffix.lower() not in EXTENSIONS:
continue
full_path = Path(dirpath) / filename
if "_av1" in full_path.stem:
continue
files.append(full_path)
return files
def main():
parser = argparse.ArgumentParser(
description="Optimize video library to AV1 using VMAF targeting."
)
parser.add_argument("directory", help="Root directory to scan")
parser.add_argument(
"--vmaf",
type=float,
default=DEFAULT_VMAF,
help=f"Target VMAF score (default: {DEFAULT_VMAF})",
)
parser.add_argument(
"--preset",
type=int,
default=DEFAULT_PRESET,
help=f"SVT-AV1 Preset (default: {DEFAULT_PRESET})",
)
parser.add_argument(
"--workers",
type=int,
default=DEFAULT_WORKERS,
help=f"Concurrent files to process (default: {DEFAULT_WORKERS})",
)
parser.add_argument(
"--samples",
type=int,
default=DEFAULT_SAMPLES,
help=f"Samples to use for CRF search if supported (default: {DEFAULT_SAMPLES})",
)
parser.add_argument(
"--thorough",
action="store_true",
help="Use ab-av1 thorough mode if supported (slower, more accurate)",
)
parser.add_argument(
"--encoder",
default="svt-av1",
help="ab-av1 encoder (default: svt-av1). For AMD AV1 on Windows try: av1_amf",
)
parser.add_argument(
"--hwaccel",
default=None,
help=(
"Hardware acceleration for decode (passed via ab-av1 --enc-input if supported). "
"Examples: auto, vaapi, d3d11va, videotoolbox. Use 'none' to disable."
),
)
parser.add_argument(
"--hwaccel-output-format",
default=None,
help="Optional hwaccel_output_format override (e.g., vaapi)",
)
args = parser.parse_args()
if args.workers < 1:
print("Error: --workers must be >= 1")
sys.exit(2)
check_dependencies()
root = Path(args.directory)
if not root.exists():
print(f"Directory not found: {root}")
sys.exit(1)
print(f"Platform: {platform_label()}")
print(f"Scanning library: {root}")
print(f"Target VMAF: {args.vmaf}")
print(f"Encoder Preset: {args.preset}")
print(f"Workers: {args.workers}")
print(f"Samples: {args.samples}")
print(f"Encoder: {args.encoder}")
if args.hwaccel is not None:
print(f"HWAccel: {args.hwaccel}")
print("-" * 50)
files = scan_library(root)
if not files:
print("No media files found.")
return
if args.workers == 1:
for file_path in files:
process_file(file_path, args)
return
with ThreadPoolExecutor(max_workers=args.workers) as executor:
futures = [
executor.submit(process_file, file_path, args) for file_path in files
]
for future in as_completed(futures):
try:
future.result()
except Exception as e:
print(f"Unexpected error: {e}")
if __name__ == "__main__":
main()
run.ps1
-12
View File
@@ -1,12 +0,0 @@
# Run Smart GPU Encoder (Wrapper)
$ScriptPath = "$PSScriptRoot\smart_gpu_encoder.py"
$PythonPath = "python"
# Default to interactive mode if no args
if ($args.Count -eq 0) {
Write-Host "Starting Smart Encoder..." -ForegroundColor Cyan
& $PythonPath $ScriptPath
} else {
# Pass through arguments
& $PythonPath $ScriptPath $args
}
run_smart_encoder.sh
-191
View File
@@ -1,191 +0,0 @@
#!/bin/bash
# Smart Video Encoder Launcher - Linux/WSL
# Usage: ./run_smart_encoder.sh [options]
set -e
# Colors
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
CYAN='\033[0;36m'
MAGENTA='\033[0;35m'
RESET='\033[0m'
# Configuration
TV_DIR="${TV_DIR:-/mnt/z/tv}"
CONTENT_DIR="${CONTENT_DIR:-/mnt/z/content}"
JOBS="${JOBS:-1}"
TV_ONLY=false
CONTENT_ONLY=false
# Parse arguments
while [[ $# -gt 0 ]]; do
case "$1" in
--tv-dir)
TV_DIR="$2"
shift 2
;;
--content-dir)
CONTENT_DIR="$2"
shift 2
;;
--jobs)
JOBS="$2"
shift 2
;;
--tv-only)
TV_ONLY=true
shift
;;
--content-only)
CONTENT_ONLY=true
shift
;;
--help|-h)
echo "Usage: $0 [options]"
echo ""
echo "Options:"
echo " --tv-dir <path> TV directory (default: /mnt/z/tv)"
echo " --content-dir <path> Content directory (default: /mnt/z/content)"
echo " --jobs <count> Parallel jobs (default: 1)"
echo " --tv-only Process TV directory only"
echo " --content-only Process content directory only"
echo " --help, -h Show this help"
exit 0
;;
*)
echo "Unknown option: $1"
echo "Use --help for usage"
exit 1
;;
esac
done
log_info() {
echo -e "${CYAN}$*${RESET}"
}
log_success() {
echo -e "${GREEN}✓ $*${RESET}"
}
log_error() {
echo -e "${RED}❌ $*${RESET}"
}
log_warn() {
echo -e "${YELLOW}⚠️ $*${RESET}"
}
# Get script directory
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PYTHON_SCRIPT="$SCRIPT_DIR/smart_encoder.py"
# Check dependencies
log_info "Checking dependencies..."
if ! command -v python3 &> /dev/null; then
log_error "Python 3 not found"
log_warn "Install with: sudo apt install python3"
exit 1
fi
if ! command -v ffmpeg &> /dev/null; then
log_error "FFmpeg not found"
log_warn "Install with: sudo apt install ffmpeg"
exit 1
fi
if ! command -v ffprobe &> /dev/null; then
log_error "FFprobe not found"
log_warn "Install with: sudo apt install ffmpeg"
exit 1
fi
if ! command -v ab-av1 &> /dev/null; then
log_error "ab-av1 not found"
log_warn "Install with: cargo install ab-av1"
exit 1
fi
log_success "All dependencies found"
# Check if script exists
if [[ ! -f "$PYTHON_SCRIPT" ]]; then
log_error "smart_encoder.py not found at: $PYTHON_SCRIPT"
exit 1
fi
# Make script executable
chmod +x "$PYTHON_SCRIPT"
# Print banner
echo ""
echo -e "${MAGENTA}============================================================${RESET}"
echo -e "${MAGENTA}🎬 Smart Video Encoder - Linux/WSL${RESET}"
echo -e "${MAGENTA}============================================================${RESET}"
echo ""
# Print configuration
log_info "Configuration:"
echo " TV Directory: $TV_DIR"
echo " Content Directory: $CONTENT_DIR"
echo " Parallel Jobs: $JOBS"
if [[ "$TV_ONLY" == true ]]; then
echo -e " ${YELLOW}Mode: TV only${RESET}"
elif [[ "$CONTENT_ONLY" == true ]]; then
echo -e " ${YELLOW}Mode: Content only${RESET}"
else
echo -e " ${YELLOW}Mode: TV + Content${RESET}"
fi
# Check directories
if [[ "$TV_ONLY" == false ]] && [[ ! -d "$TV_DIR" ]]; then
log_warn "TV directory not found: $TV_DIR"
fi
if [[ "$CONTENT_ONLY" == false ]] && [[ ! -d "$CONTENT_DIR" ]]; then
log_warn "Content directory not found: $CONTENT_DIR"
fi
# Build command
CMD="python3 $PYTHON_SCRIPT --tv-dir $TV_DIR --content-dir $CONTENT_DIR --jobs $JOBS"
if [[ "$TV_ONLY" == true ]]; then
CMD="$CMD --tv-only"
elif [[ "$CONTENT_ONLY" == true ]]; then
CMD="$CMD --content-only"
fi
# Run
echo ""
log_info "Starting encoder..."
echo -e "${CYAN}============================================================${RESET}"
echo ""
eval $CMD
EXIT_CODE=$?
echo ""
echo -e "${CYAN}============================================================${RESET}"
if [[ $EXIT_CODE -eq 0 ]]; then
log_success "Encoding completed successfully"
else
log_error "Encoding failed with exit code: $EXIT_CODE"
fi
echo -e "${CYAN}============================================================${RESET}"
# Show log locations
echo ""
log_info "Log files:"
echo " $HOME/Videos/encodes/logs/tv.jsonl"
echo " $HOME/Videos/encodes/logs/content.jsonl"
echo " $HOME/Videos/encodes/logs/rejected.jsonl"
echo " $HOME/Videos/encodes/logs/metadata.jsonl"
echo ""
exit $EXIT_CODE
src/smart_encoder.py
-444
View File
@@ -1,444 +0,0 @@
#!/usr/bin/env python3
"""
Smart Video Encoder - AV1 with HEVC Fallback
Runs on Windows with native ab-av1.exe and ffmpeg
Refactored to use vmaf_common.py
"""
import os
import sys
import subprocess
import json
import shutil
import time
import argparse
import signal
import platform
import threading
from pathlib import Path
from concurrent.futures import ThreadPoolExecutor, as_completed
import vmaf_common as common
# --- Configuration ---
TARGET_VMAF_MIN = common.DEFAULT_CONFIG["target_vmaf"]
MIN_SAVINGS_PERCENT = common.DEFAULT_CONFIG["min_savings"]
MAX_JOBS = common.DEFAULT_CONFIG["cpu_jobs"]
AV1_QUALITY = common.DEFAULT_CONFIG["av1_crf"]
HEVC_QUALITY = common.DEFAULT_CONFIG["hevc_crf"]
SAMPLE_DURATION = 60
SAMPLE_START = 300
MAX_SAMPLE_SIZE_MB = 150
TEMP_DIR = common.get_temp_dir()
# Tools
FFMPEG_BIN = "ffmpeg"
AB_AV1_BIN = "ab-av1"
# Global state
_shutdown_requested = False
_lock_files = {}
def signal_handler(signum, frame):
global _shutdown_requested
_shutdown_requested = True
print("\n\n[WARNING] Shutdown requested. Finishing current tasks...")
if platform.system() != "Windows":
signal.signal(signal.SIGINT, signal_handler)
signal.signal(signal.SIGTERM, signal_handler)
def run_command_streaming(cmd, description=""):
"""Run command and stream output in real-time"""
print(f" ▶ Running: {description}")
process = subprocess.Popen(
cmd,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
universal_newlines=True,
bufsize=1
)
output_lines = []
if process.stdout:
for line in process.stdout:
if _shutdown_requested:
process.terminate()
break
line = line.rstrip()
print(f" {line}")
output_lines.append(line)
process.wait()
return process.returncode, "\n".join(output_lines)
def test_av1_sample(filepath, output_path):
"""Test AV1 encoding on a sample segment"""
print(f" 🧪 Testing AV1 with {SAMPLE_DURATION}s sample...")
cmd = [
"ffmpeg",
"-hide_banner",
"-loglevel", "warning",
"-ss", str(SAMPLE_START),
"-i", str(filepath),
"-t", str(SAMPLE_DURATION),
"-c:v", "libsvtav1",
"-crf", str(AV1_QUALITY),
"-preset", "6",
"-c:a", "copy",
"-c:s", "copy",
"-y",
str(output_path)
]
returncode, output = run_command_streaming(cmd, f"AV1 sample test")
return returncode == 0 and output_path.exists()
def encode_av1_full(filepath, output_path):
"""Full AV1 encode using ab-av1"""
print(f" 🎬 Full AV1 encode...")
cmd = [
"ab-av1", "encode",
"-i", str(filepath),
"-o", str(output_path),
"--crf", str(AV1_QUALITY),
"--preset", "6",
"--acodec", "copy"
]
return run_command_streaming(cmd, f"AV1 full encode")[0] == 0
def encode_hevc_full(filepath, output_path):
"""Full HEVC encode using ffmpeg"""
print(f" 🎬 Full HEVC encode...")
cmd = [
"ffmpeg",
"-hide_banner",
"-loglevel", "warning",
"-i", str(filepath),
"-c:v", "libx265",
"-crf", str(HEVC_QUALITY),
"-preset", "medium",
"-c:a", "copy",
"-c:s", "copy",
"-y",
str(output_path)
]
return run_command_streaming(cmd, f"HEVC full encode")[0] == 0
def process_file(filepath, log_dir, log_category, lock_dir):
"""Process a single video file with AV1→HEVC fallback"""
filepath = Path(filepath)
filename = filepath.name
# Check lock (using shared logic now)
lock_file = common.acquire_lock(lock_dir, filepath, None) # Legacy encoder, no category root
if not lock_file:
print(f" 🔒 Skipping (locked): {filename}")
return True
try:
print(f"\n{'='*60}")
print(f"📁 Processing: {filename}")
print(f"{'='*60}")
# Get initial metadata
metadata_before = common.get_video_info(filepath)
if not metadata_before:
print(f"❌ Could not read metadata, skipping")
return False
print(f" 📊 Original:")
print(f" Codec: {metadata_before['codec']}")
print(f" Size: {metadata_before['size'] / (1024**3):.2f} GB")
print(f" Bitrate: {metadata_before['bitrate']} kbps")
print(f" Duration: {metadata_before['duration'] / 60:.1f} min")
# Skip if already AV1 or HEVC
if metadata_before['codec'] in ['av1', 'hevc']:
print(f" ℹ️ Already optimized ({metadata_before['codec']}), skipping")
# We don't log skips to main log to avoid clutter, but could if needed
return True
input_size = metadata_before['size']
# --- PHASE 1: AV1 SAMPLE TEST ---
sample_output = TEMP_DIR / f"{filepath.stem}.sample.mkv"
use_av1 = True
try:
av1_test_passed = test_av1_sample(filepath, sample_output)
if av1_test_passed:
sample_size = sample_output.stat().st_size
sample_size_mb = sample_size / (1024 * 1024)
print(f" 📏 Sample size: {sample_size_mb:.1f} MB")
# Extrapolate to full file size
duration_ratio = metadata_before['duration'] / SAMPLE_DURATION
estimated_full_size = sample_size * duration_ratio
estimated_full_gb = estimated_full_size / (1024**3)
input_gb = input_size / (1024**3)
print(f" 📈 Estimated full size: {estimated_full_gb:.2f} GB")
print(f" 📉 Original size: {input_gb:.2f} GB")
if sample_size_mb > MAX_SAMPLE_SIZE_MB:
print(f" ❌ AV1 REJECTED: Sample too large ({sample_size_mb:.1f} MB > {MAX_SAMPLE_SIZE_MB} MB)")
use_av1 = False
elif estimated_full_size >= input_size:
print(f" ❌ AV1 REJECTED: Estimated size ({estimated_full_gb:.2f} GB) >= original ({input_gb:.2f} GB)")
use_av1 = False
else:
estimated_savings = (1 - estimated_full_size / input_size) * 100
print(f" ✅ AV1 PASS: Estimated savings {estimated_savings:.1f}%")
else:
print(f" ❌ AV1 sample test failed")
use_av1 = False
if sample_output.exists():
sample_output.unlink()
except Exception as e:
print(f" ❌ AV1 test error: {e}")
use_av1 = False
# --- PHASE 2: ENCODE ---
temp_output = TEMP_DIR / f"{filepath.stem}.temp.mkv"
final_status = "rejected"
final_codec = None
final_size = input_size
final_savings = 0.0
if use_av1:
# Try AV1 full encode
if encode_av1_full(filepath, temp_output):
metadata_after = common.get_video_info(temp_output)
if metadata_after:
final_size = metadata_after['size']
final_savings = (1 - final_size / input_size) * 100
if final_size < input_size:
final_status = "success"
final_codec = "av1"
print(f" ✅ AV1 SUCCESS: Saved {final_savings:.1f}%")
else:
print(f" ❌ AV1 FAILED: Final size >= original")
if temp_output.exists(): temp_output.unlink()
# Fall back to HEVC
print(f" 🔄 Trying HEVC fallback...")
if encode_hevc_full(filepath, temp_output):
metadata_after = common.get_video_info(temp_output)
if metadata_after:
final_size = metadata_after['size']
final_savings = (1 - final_size / input_size) * 100
if final_size < input_size:
final_status = "success"
final_codec = "hevc"
print(f" ✅ HEVC SUCCESS: Saved {final_savings:.1f}%")
else:
print(f" ❌ HEVC FAILED: Also larger than original")
if temp_output.exists(): temp_output.unlink()
else:
print(f" ❌ AV1 encode failed, trying HEVC...")
if encode_hevc_full(filepath, temp_output):
metadata_after = common.get_video_info(temp_output)
if metadata_after:
final_size = metadata_after['size']
final_savings = (1 - final_size / input_size) * 100
if final_size < input_size:
final_status = "success"
final_codec = "hevc"
print(f" ✅ HEVC SUCCESS: Saved {final_savings:.1f}%")
else:
print(f" ❌ HEVC FAILED: Larger than original")
if temp_output.exists(): temp_output.unlink()
else:
# AV1 test failed, try HEVC directly
print(f" 🔄 Trying HEVC directly...")
if encode_hevc_full(filepath, temp_output):
metadata_after = common.get_video_info(temp_output)
if metadata_after:
final_size = metadata_after['size']
final_savings = (1 - final_size / input_size) * 100
if final_size < input_size:
final_status = "success"
final_codec = "hevc"
print(f" ✅ HEVC SUCCESS: Saved {final_savings:.1f}%")
else:
print(f" ❌ HEVC FAILED: Larger than original")
if temp_output.exists(): temp_output.unlink()
# --- PHASE 3: FINALIZE ---
if final_status == "success":
# Replace original file (Safe Upload)
if filepath.suffix:
backup_path = filepath.with_suffix(f"{filepath.suffix}.backup")
else:
backup_path = Path(str(filepath) + ".backup")
shutil.move(str(filepath), str(backup_path))
shutil.move(str(temp_output), str(filepath))
# Verify Integrity
if Path(filepath).stat().st_size == final_size:
backup_path.unlink()
metadata_after = common.get_video_info(filepath)
common.log_event(log_dir, f"{log_category}.jsonl", {
"file": str(filepath),
"status": "success",
"codec": final_codec,
"input_size": input_size,
"output_size": final_size,
"savings_percent": final_savings,
"metadata_before": metadata_before,
"metadata_after": metadata_after
})
print(f"\n ✅ SUCCESS: Optimized with {final_codec.upper() if final_codec else 'unknown'}")
print(f" Savings: {final_savings:.1f}% ({input_size / (1024**3):.2f} GB → {final_size / (1024**3):.2f} GB)")
else:
print(" ❌ Critical Error: Copied file size mismatch! Restoring backup.")
shutil.move(str(backup_path), str(filepath))
else:
common.log_event(log_dir, "rejected.jsonl", {
"file": str(filepath),
"status": "rejected",
"reason": "both_codecs_larger_than_original",
"input_size": input_size,
"metadata": metadata_before
})
print(f"\n ❌ REJECTED: Both AV1 and HEVC larger than original")
print(f" Keeping original file ({input_size / (1024**3):.2f} GB)")
return True
except Exception as e:
print(f"❌ Error processing {filename}: {e}")
return False
finally:
# Release shared lock
if lock_file and lock_file.exists():
lock_file.unlink()
def scan_directory(directory, extensions=None):
"""Scan directory for video files"""
if extensions is None:
extensions = {".mkv", ".mp4", ".mov", ".avi", ".ts"}
files = []
for dirpath, dirnames, filenames in os.walk(directory):
# Skip processed/system files
dirnames[:] = [d for d in dirnames if not d.startswith("_") and d not in [".recycle", "@eaDir"]]
for filename in filenames:
filepath = Path(dirpath) / filename
if filepath.suffix.lower() in extensions:
if "_av1" in filepath.stem.lower() or "_hevc" in filepath.stem.lower():
continue
files.append(filepath)
return sorted(files)
def main():
parser = argparse.ArgumentParser(description="Smart Video Encoder - AV1 with HEVC Fallback")
parser.add_argument("--tv-dir", default=common.DEFAULT_CONFIG["tv_dir"], help="TV directory")
parser.add_argument("--content-dir", default=common.DEFAULT_CONFIG["content_dir"], help="Content directory")
parser.add_argument("--jobs", type=int, default=MAX_JOBS, help="Parallel jobs")
parser.add_argument("--tv-only", action="store_true", help="Process TV only")
parser.add_argument("--content-only", action="store_true", help="Process content only")
args = parser.parse_args()
print("="*60)
print("🎬 Smart Video Encoder - AV1 with HEVC Fallback")
print("="*60)
# Setup
common.check_dependencies()
lock_dir, log_dir = common.get_base_paths(args)
TEMP_DIR.mkdir(parents=True, exist_ok=True)
print(f"\n📁 Configuration:")
print(f" TV Directory: {args.tv_dir}")
print(f" Content Directory: {args.content_dir}")
print(f" Parallel Jobs: {args.jobs}")
print(f" Locks: {lock_dir}")
print(f" Logs: {log_dir}")
print()
tasks = []
if not args.content_only:
tv_dir = Path(args.tv_dir)
if tv_dir.exists():
tv_files = scan_directory(tv_dir)
print(f"📺 TV Files: {len(tv_files)}")
for f in tv_files:
tasks.append((f, log_dir, "tv_shows", lock_dir))
if not args.tv_only:
content_dir = Path(args.content_dir)
if content_dir.exists():
content_files = scan_directory(content_dir)
print(f"📦 Content Files: {len(content_files)}")
for f in content_files:
tasks.append((f, log_dir, "content", lock_dir))
if not tasks:
print("❌ No files to process")
return
print(f"\n🚀 Processing {len(tasks)} files...")
print("="*60)
success_count = 0
fail_count = 0
with ThreadPoolExecutor(max_workers=args.jobs) as executor:
futures = {}
for item in tasks:
# item = (filepath, log_dir, log_category, lock_dir)
future = executor.submit(process_file, *item)
futures[future] = item[0]
for future in as_completed(futures):
if _shutdown_requested:
break
filepath = futures[future]
try:
result = future.result()
if result:
success_count += 1
else:
fail_count += 1
except Exception as e:
print(f"❌ Error processing {filepath}: {e}")
fail_count += 1
print("\n" + "="*60)
print("📊 Summary:")
print(f" Processed: {success_count + fail_count}")
print(f" ✅ Success: {success_count}")
print(f" ❌ Failed: {fail_count}")
print("="*60)
if __name__ == "__main__":
main()
src/smart_gpu_encoder.py
-494
View File
@@ -1,494 +0,0 @@
#!/usr/bin/env python3
"""
Smart GPU Encoder (Windows/AMD/NVIDIA Optimized)
------------------------------------------------
Refactored to use vmaf_common.py
"""
import os
import sys
import subprocess
import json
import shutil
import time
import argparse
import signal
import platform
import re
import threading
from pathlib import Path
from datetime import datetime
from concurrent.futures import ThreadPoolExecutor, as_completed
# --- Import Common Module ---
import vmaf_common as common
# --- Configuration ---
TARGET_VMAF_MIN = common.DEFAULT_CONFIG["target_vmaf"]
MIN_SAVINGS_PERCENT = common.DEFAULT_CONFIG["min_savings"]
MAX_JOBS = common.DEFAULT_CONFIG["gpu_jobs"]
DEFAULT_AV1_QP = 32
DEFAULT_HEVC_QP = 28
SAMPLE_DURATION = 60
SAMPLE_START_TIME = 300
TEMP_DIR = None # Will be set in main
# Tools (Local override if needed, else used from common checks)
FFMPEG_BIN = "ffmpeg"
AB_AV1_BIN = "ab-av1"
# Global state
shutdown_requested = False
active_processes = set()
upload_lock = threading.Lock()
proc_lock = threading.Lock()
debug_mode = False
def handle_sigint(signum, frame):
global shutdown_requested
print("\n\n[!] CRITICAL: Shutdown requested (Ctrl+C).")
print(" Killing active encoder processes...")
shutdown_requested = True
with proc_lock:
for proc in list(active_processes):
try:
proc.terminate()
time.sleep(0.1)
if proc.poll() is None:
proc.kill()
except:
pass
print(" Cleanup complete. Exiting.")
sys.exit(1)
signal.signal(signal.SIGINT, handle_sigint)
# --- Hardware Detection ---
def detect_hardware_encoder():
"""Detects available hardware encoders via ffmpeg"""
# Use common module detection
return common.detect_hardware_encoder()
def get_encoder_args(codec, encoder, qp):
"""Returns correct ffmpeg args for specific HW vendor"""
# Use common module args
return common.get_encoder_args(codec, encoder, qp)
# --- Helpers ---
def run_process(cmd, description="", status_callback=None):
"""Run a process with real-time output and clean shutdown tracking"""
if shutdown_requested: return False
if status_callback: status_callback(description)
try:
# Windows: Hide console window
cflags = 0x08000000 if platform.system() == 'Windows' else 0
proc = subprocess.Popen(
cmd,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
universal_newlines=True,
bufsize=1,
creationflags=cflags
)
with proc_lock:
active_processes.add(proc)
if proc.stdout:
for line in proc.stdout:
if shutdown_requested:
proc.terminate()
break
line = line.strip()
if line:
if debug_mode: print(f" [Debug] {line}")
if status_callback and ("frame=" in line or "size=" in line or "time=" in line):
status_callback(line)
proc.wait()
with proc_lock:
if proc in active_processes:
active_processes.remove(proc)
return proc.returncode == 0
except Exception as e:
if status_callback: status_callback(f"Error: {e}")
return False
def run_vmaf_check(reference, distorted, status_callback=None):
"""Run ab-av1 vmaf to get score"""
# Use common dependency check to find binary if needed, but here just assume it's in path or bin
ab_exe = "ab-av1"
# Check if bundled exists
bundled = Path(__file__).parent / "bin" / "ab-av1.exe"
if bundled.exists():
ab_exe = str(bundled)
cmd = [ab_exe, "vmaf", "--reference", str(reference), "--distorted", str(distorted)]
if status_callback: status_callback("Calculating VMAF...")
try:
result = subprocess.run(cmd, capture_output=True, text=True, check=True)
for line in result.stdout.splitlines():
line = line.strip()
match = re.search(r"VMAF\s+([0-9.]+)", line)
if match: return float(match.group(1))
try:
val = float(line)
if 0 <= val <= 100: return val
except: pass
return 0.0
except Exception:
return -1.0
# --- Core Logic ---
def process_file(filepath, log_category, lock_dir, log_dir, encoders, category_root=None, worker_id=0, status_cb=None):
"""
Process a single file.
status_cb: function(worker_id, filename, status_text, color)
category_root: Root directory (tv_dir or content_dir) for relative path calculation
"""
av1_enc, hevc_enc, hw_type = encoders
filepath = Path(filepath)
filename = filepath.name
def update(msg, color="white"):
if status_cb: status_cb(worker_id, filename, msg, color)
else: print(f"[{worker_id}] {msg}")
if shutdown_requested: return
# 0. Check if already processed in a previous run
if common.is_already_processed(log_dir, filepath):
update("Already Processed (Skipping)", "dim")
return
# 1. Lock Check (Shared Storage)
lock_file = common.acquire_lock(lock_dir, filepath, category_root)
if not lock_file:
return # Locked or skipped
try:
update("Analyzing...", "blue")
# 2. Analyze Source
info = common.get_video_info(filepath)
if not info:
update("Metadata Error", "red")
return
if info["codec"] == "av1":
update("Already AV1 (Skipping)", "green")
time.sleep(1)
return
# 3. Create Samples
sample_ref = TEMP_DIR / f"{filepath.stem}_{worker_id}_ref.mkv"
sample_enc = TEMP_DIR / f"{filepath.stem}_{worker_id}_enc.mkv"
sample_start = SAMPLE_START_TIME
if info["duration"] < (SAMPLE_START_TIME + SAMPLE_DURATION):
sample_start = max(0, (info["duration"] / 2) - (SAMPLE_DURATION / 2))
update("Extracting Ref", "magenta")
cmd_ref = [
FFMPEG_BIN, "-y", "-hide_banner", "-loglevel", "error",
"-ss", str(sample_start), "-t", str(SAMPLE_DURATION),
"-i", str(filepath), "-c", "copy", "-map", "0:v:0",
str(sample_ref)
]
if not run_process(cmd_ref):
update("Extract Ref Failed", "red")
return
# TEST 1: AV1
vmaf_score = 0
savings = 0
if av1_enc:
update(f"Testing AV1 QP{DEFAULT_AV1_QP}", "yellow")
enc_args = get_encoder_args("av1", av1_enc, DEFAULT_AV1_QP)
cmd_enc = [
FFMPEG_BIN, "-y", "-hide_banner", "-loglevel", "error",
"-i", str(sample_ref), *enc_args, "-an", str(sample_enc)
]
if run_process(cmd_enc):
update("Calculating VMAF", "cyan")
vmaf_score = run_vmaf_check(sample_ref, sample_enc)
ref_size = sample_ref.stat().st_size
enc_size = sample_enc.stat().st_size
savings = (1 - (enc_size / ref_size)) * 100 if ref_size > 0 else 0
else:
update("AV1 Test Failed", "red")
chosen_codec = None
chosen_qp = 0
# Decision Logic
if vmaf_score >= TARGET_VMAF_MIN and savings >= MIN_SAVINGS_PERCENT:
chosen_codec = "av1"
chosen_qp = DEFAULT_AV1_QP
update(f"AV1 Good (VMAF {vmaf_score:.1f})", "green")
# Smart Optimization
if vmaf_score > 97.0:
update(f"Optimizing (High Quality {vmaf_score:.1f})", "yellow")
new_qp = DEFAULT_AV1_QP + 4
args_opt = get_encoder_args("av1", av1_enc, new_qp)
cmd_opt = [FFMPEG_BIN, "-y", "-hide_banner", "-loglevel", "error", "-i", str(sample_ref), *args_opt, "-an", str(sample_enc)]
if run_process(cmd_opt):
vmaf_opt = run_vmaf_check(sample_ref, sample_enc)
size_opt = sample_enc.stat().st_size
sav_opt = (1 - (size_opt / sample_ref.stat().st_size)) * 100
if vmaf_opt >= TARGET_VMAF_MIN and sav_opt > savings:
update(f"Opt Accepted (+{sav_opt - savings:.1f}%)", "green")
chosen_qp = new_qp
vmaf_score = vmaf_opt
savings = sav_opt
else:
update("Testing HEVC Fallback", "magenta")
if info["codec"] != "hevc" and hevc_enc:
hevc_args = get_encoder_args("hevc", hevc_enc, DEFAULT_HEVC_QP)
cmd_hevc = [FFMPEG_BIN, "-y", "-hide_banner", "-loglevel", "error", "-i", str(sample_ref), *hevc_args, "-an", str(sample_enc)]
run_process(cmd_hevc)
vmaf_score = run_vmaf_check(sample_ref, sample_enc)
enc_size = sample_enc.stat().st_size
savings = (1 - (enc_size / sample_ref.stat().st_size)) * 100
if vmaf_score >= TARGET_VMAF_MIN and savings >= MIN_SAVINGS_PERCENT:
update(f"HEVC Accepted (VMAF {vmaf_score:.1f})", "green")
chosen_codec = "hevc"
chosen_qp = DEFAULT_HEVC_QP
else:
update("HEVC Rejected", "red")
common.log_event(log_dir, "rejected.jsonl", {"file": str(filepath), "status": "rejected", "vmaf": vmaf_score})
else:
update("Skipping HEVC", "yellow")
# Cleanup Samples
if sample_ref.exists(): sample_ref.unlink()
if sample_enc.exists(): sample_enc.unlink()
# 4. Full Encode
if chosen_codec:
update(f"Encoding {chosen_codec.upper()} (QP {chosen_qp})", "green")
output_file = TEMP_DIR / f"{filepath.stem}.{chosen_codec}.mkv"
final_args = get_encoder_args(chosen_codec, av1_enc if chosen_codec=="av1" else hevc_enc, chosen_qp)
cmd_full = [
FFMPEG_BIN, "-y", "-hide_banner", "-loglevel", "info", "-stats",
"-i", str(filepath),
*final_args,
"-c:a", "copy", "-c:s", "copy", "-map", "0",
str(output_file)
]
def prog_cb(msg):
if "frame=" in msg:
try:
if "time=" in msg:
t_str = msg.split("time=")[1].split(" ")[0]
h, m, s = map(float, t_str.split(':'))
cur_sec = h*3600 + m*60 + s
percent = (cur_sec / info["duration"]) * 100
else:
percent = 0.0
speed = "1x"
if "speed=" in msg:
speed = msg.split("speed=")[1].split("x")[0] + "x"
update(f"Encoding {chosen_codec} | {percent:.1f}% | {speed}", "green")
except:
pass
if run_process(cmd_full, f"Full Encode ({chosen_codec.upper()} QP {chosen_qp})", status_callback=prog_cb):
final_info = common.get_video_info(output_file)
if not final_info: final_info = {"size": output_file.stat().st_size}
final_size = final_info["size"]
final_savings = (1 - (final_size / info["size"])) * 100
saved_bytes = info["size"] - final_size
update(f"Uploading (Saved {final_savings:.1f}%)", "blue")
# UPLOAD (Serialized)
with upload_lock:
update(f"Uploading...", "blue")
backup_path = filepath.with_suffix(f"{filepath.suffix}.original")
try:
shutil.move(str(filepath), str(backup_path))
shutil.copy2(str(output_file), str(filepath))
# Verify integrity
if Path(filepath).stat().st_size == final_size:
output_file.unlink()
backup_path.unlink()
# Refresh metadata for accuracy
final_info_verified = common.get_video_info(filepath)
common.log_event(log_dir, f"{log_category}.jsonl", {
"file": str(filepath),
"status": "success",
"codec": chosen_codec,
"vmaf": vmaf_score,
"savings": final_savings,
"original_metadata": info,
"encoded_metadata": final_info_verified or final_info
})
# Mark as processed to prevent re-encoding in future runs
common.mark_processed(log_dir, filepath, chosen_codec, vmaf_score, final_savings)
# Mark lock as completed (keep it for future runs)
common.mark_lock_completed(lock_file)
update("Done", "green")
if status_cb: status_cb(worker_id, filename, f"STATS:SAVED:{saved_bytes}", "green")
else:
update("Upload Failed (Size Mismatch)", "red")
shutil.move(str(backup_path), str(filepath))
except Exception as e:
update(f"Move Error: {str(e)[:20]}", "red")
if backup_path.exists(): shutil.move(str(backup_path), str(filepath))
else:
update("Encode Failed", "red")
if output_file.exists(): output_file.unlink()
except Exception as e:
update(f"Error: {str(e)[:30]}", "red")
# On error, delete lock so file can be retried
if lock_file and lock_file.exists():
lock_file.unlink()
finally:
update("Idle", "dim")
def main():
global debug_mode
parser = argparse.ArgumentParser()
parser.add_argument("--tv-dir", default=common.DEFAULT_CONFIG["tv_dir"])
parser.add_argument("--content-dir", default=common.DEFAULT_CONFIG["content_dir"])
parser.add_argument("--jobs", type=int, default=MAX_JOBS)
parser.add_argument("--debug", action="store_true")
parser.add_argument("--skip-until", help="Skip all files alphabetically until this filename substring is found")
parser.add_argument("--cpu-only", action="store_true", help="Force software encoding (CPU only)")
parser.add_argument("--temp-dir", help="Override local temp directory")
parser.add_argument("--av1-encoder", choices=["hw", "sw", "off"], default="hw", help="AV1 encoder: hw (hardware), sw (software), off (disable)")
parser.add_argument("--hevc-encoder", choices=["hw", "sw", "off"], default="hw", help="HEVC encoder: hw (hardware), sw (software), off (disable)")
args = parser.parse_args()
if args.debug:
debug_mode = True
print("[Debug Mode Enabled]")
# 0. Check Dependencies
common.check_dependencies()
# 1. Setup Directories
lock_dir, log_dir = common.get_base_paths(args)
global TEMP_DIR
TEMP_DIR = common.get_temp_dir(args)
# 2. Detect Hardware
av1, hevc, hw = common.detect_hardware_encoder(args)
print("="*60)
print(f" SMART ENCODER | Hardware: {hw.upper()} | Jobs: {args.jobs}")
if hw == "cpu":
print(f" [!] Fallback to CPU Software Encoding (Slow)")
if av1: print(f" AV1: {av1} (libsvtav1)")
if hevc: print(f" HEVC: {hevc} (libx265)")
else:
print(f" AV1: {av1} | HEVC: {hevc}")
print(f" Locks: {lock_dir}")
print("="*60)
# 3. Scan & Queue
tasks = []
# Skip-until filtering
skip_until = args.skip_until
skipping = bool(skip_until)
skipped_count = 0
tv_path = Path(args.tv_dir)
if tv_path.exists():
print(f"Scanning TV: {tv_path}")
files = list(tv_path.rglob("*.mkv")) + list(tv_path.rglob("*.mp4"))
files.sort() # Alphabetical order for consistency across platforms
for f in files:
if skipping:
if skip_until.lower() in str(f).lower():
skipping = False
print(f" Found '{skip_until}' - resuming from here")
else:
skipped_count += 1
continue
tasks.append((f, "tv_shows"))
content_path = Path(args.content_dir)
if content_path.exists():
print(f"Scanning Content: {content_path}")
files = list(content_path.rglob("*.mkv")) + list(content_path.rglob("*.mp4"))
files.sort() # Alphabetical order for consistency across platforms
for f in files:
if skipping:
if skip_until.lower() in str(f).lower():
skipping = False
print(f" Found '{skip_until}' - resuming from here")
else:
skipped_count += 1
continue
tasks.append((f, "content"))
if skipped_count > 0:
print(f" Skipped {skipped_count} files (--skip-until)")
if not tasks:
print("No files found.")
return
# 4. Execute
print(f"\n🚀 Processing {len(tasks)} files...")
# Build category root map
category_roots = {
"tv_shows": Path(args.tv_dir),
"content": Path(args.content_dir)
}
with ThreadPoolExecutor(max_workers=args.jobs) as executor:
futures = {
executor.submit(process_file, f, cat, lock_dir, log_dir, (av1, hevc, hw), category_roots.get(cat)): f
for f, cat in tasks
}
for future in as_completed(futures):
if shutdown_requested:
executor.shutdown(wait=False, cancel_futures=True)
break
try:
future.result()
except Exception as e:
print(f"Worker Error: {e}")
if __name__ == "__main__":
main()
src/smart_monitor.py
-427
View File
@@ -1,427 +0,0 @@
#!/usr/bin/env python3
r"""
Smart VMAF Monitor & Dashboard
------------------------------
The main interface for the VMAF Optimizer.
Provides a TUI (Text User Interface) to visualize encoding progress.
Usage:
python smart_monitor.py --tv-dir "Z:\tv" --content-dir "Z:\content" --jobs 4 [--monitor]
"""
import os
import time
import argparse
import sys
import threading
import queue
import json
import re
from pathlib import Path
# Import the engine
import smart_gpu_encoder as encoder
import vmaf_common as common
# UI Library
try:
from rich.console import Console
from rich.live import Live
from rich.table import Table
from rich.layout import Layout
from rich.panel import Panel
from rich.text import Text
HAS_RICH = True
except ImportError:
HAS_RICH = False
print("Warning: 'rich' library not found. Running in basic mode.")
# Watchdog
try:
from watchdog.observers import Observer
from watchdog.events import FileSystemEventHandler
HAS_WATCHDOG = True
except ImportError:
HAS_WATCHDOG = False
# --- Caching ---
CACHE_FILE = Path("library_cache.json")
def load_cache():
if CACHE_FILE.exists():
try:
with open(CACHE_FILE, "r") as f:
return set(json.load(f))
except: pass
return set()
def save_cache(files):
try:
with open(CACHE_FILE, "w") as f:
# Convert paths to strings for JSON
json.dump([str(f) for f in files], f)
except: pass
def fast_scan(path):
"""Recursive scan using scandir (faster than pathlib)"""
files = []
try:
if not os.path.exists(path): return []
for entry in os.scandir(path):
if entry.is_dir():
files.extend(fast_scan(entry.path))
elif entry.is_file():
if entry.name.lower().endswith(('.mkv', '.mp4')):
if "_enc" not in entry.name and "_ref" not in entry.name:
files.append(entry.path)
except:
pass
# Sort alphabetically for consistent ordering across platforms
return sorted(files)
# --- UI State ---
class Dashboard:
def __init__(self, num_workers, log_dir=None):
self.num_workers = num_workers
self.worker_status = {i: {"file": "Idle", "action": "Waiting", "progress": 0, "speed": "", "color": "dim"} for i in range(num_workers)}
self.stats = {"processed": 0, "skipped": 0, "failed": 0, "rejected": 0, "savings_gb": 0.0}
self.recent_completed = []
self.lock = threading.Lock()
self.log_dir = log_dir
self.activity_log_file = None
# Open activity log file for streaming
if log_dir:
try:
log_path = Path(log_dir) / "activity.log"
self.activity_log_file = open(log_path, "a", encoding="utf-8")
except Exception as e:
print(f"[Warning] Could not open activity log: {e}")
def format_filename(self, filename):
# Clean Sonarr format: {Series} - S{s}E{e} - {Title} {Quality}
# Goal: Series S01E01 [Quality]
try:
# Match S01E01
match = re.search(r"(.*?) - (S\d+E\d+) - .*? ((?:Bluray|WebDL|Remux|2160p|1080p|Proper).*)\.mkv", filename, re.IGNORECASE)
if match:
series = match.group(1)[:15] # Truncate series name
s_e = match.group(2)
quality = match.group(3).split()[0] # Just take first part of quality (Bluray-2160p)
return f"{series} {s_e} [{quality}]"
# Fallback for simpler names
if len(filename) > 35:
return filename[:15] + "..." + filename[-15:]
return filename
except:
return filename
def update_worker(self, worker_id, file, action, color="blue"):
with self.lock:
display_name = self.format_filename(file)
progress = 0
speed = ""
# Parse rich status: "Encoding AV1 | 45.2% | 2.3x"
if "|" in action:
parts = action.split("|")
action_text = parts[0].strip()
for p in parts[1:]:
p = p.strip()
if "%" in p:
try: progress = float(p.replace("%", ""))
except: pass
elif "x" in p or "MB/s" in p:
speed = p
action = action_text
self.worker_status[worker_id] = {
"file": display_name,
"action": action,
"progress": progress,
"speed": speed,
"color": color
}
def add_log(self, message):
with self.lock:
ts = time.strftime("%Y-%m-%d %H:%M:%S")
ts_short = time.strftime("%H:%M:%S")
log_entry = f"[{ts_short}] {message}"
self.recent_completed.insert(0, log_entry)
if len(self.recent_completed) > 12:
self.recent_completed.pop()
# Stream to activity log file
if self.activity_log_file:
try:
self.activity_log_file.write(f"[{ts}] {message}\n")
self.activity_log_file.flush()
except:
pass
def update_stats(self, key, val=1):
with self.lock:
if key == "savings_gb": self.stats[key] += val
else: self.stats[key] += val
def get_renderable(self):
if not HAS_RICH: return ""
layout = Layout()
layout.split_column(
Layout(name="header", size=3),
Layout(name="workers", size=self.num_workers + 4),
Layout(name="stats", size=3),
Layout(name="logs", ratio=1)
)
layout["header"].update(Panel(Text("Smart GPU Encoder (AMD AMF + VMAF)", justify="center", style="bold cyan")))
# Workers Table
table = Table(box=None, expand=True, padding=(0, 1))
table.add_column("ID", width=3, style="dim", no_wrap=True)
table.add_column("File", ratio=6, no_wrap=True)
table.add_column("Action", ratio=3, no_wrap=True)
table.add_column("Progress", width=20, no_wrap=True)
table.add_column("Speed", width=12, no_wrap=True)
for i in range(self.num_workers):
ws = self.worker_status[i]
pct = ws["progress"]
# Rich Bar
if pct > 0:
bar_len = 12
filled = int(bar_len * (pct / 100))
bar_str = "━" * filled + " " * (bar_len - filled)
prog_render = Text(f"{bar_str} {pct:.1f}%", style="green")
else:
prog_render = Text("")
table.add_row(
str(i+1),
Text(ws["file"], style="white"),
Text(ws["action"], style=ws["color"]),
prog_render,
Text(ws["speed"], style="yellow")
)
layout["workers"].update(Panel(table, title="Active Workers"))
stat_str = f"Processed: [green]{self.stats['processed']}[/] | Skipped: [yellow]{self.stats['skipped']}[/] | Rejected: [magenta]{self.stats['rejected']}[/] | Failed: [red]{self.stats['failed']}[/] | Saved: [bold green]{self.stats['savings_gb']:.2f} GB[/]"
layout["stats"].update(Panel(Text.from_markup(stat_str, justify="center")))
layout["logs"].update(Panel("\n".join(self.recent_completed), title="Activity Log"))
return layout
# --- Worker Bridge ---
def worker_wrapper(worker_id, file_path, category, category_root, lock_dir, log_dir, encoders, dashboard):
def status_callback(w_id, fname, msg, color):
# Handle Stats Signal
if msg.startswith("STATS:"):
parts = msg.split(":")
if parts[1] == "SAVED":
try:
bytes_saved = float(parts[2])
dashboard.update_stats("savings_gb", bytes_saved / (1024**3))
dashboard.update_stats("processed")
except: pass
return
# Update Live Table
dashboard.update_worker(worker_id, fname, msg, color)
# Check for Completion Events to update History
# Match keywords from smart_gpu_encoder.py
if "Done" in msg:
dashboard.add_log(f"[Success] {dashboard.format_filename(fname)}")
elif "Skipped" in msg or "Skipping" in msg:
dashboard.add_log(f"[Skip] {dashboard.format_filename(fname)}")
dashboard.update_stats("skipped")
elif "Rejected" in msg:
dashboard.add_log(f"[Reject] {dashboard.format_filename(fname)}")
dashboard.update_stats("rejected")
# Rejected doesn't count as failed, just processed (or ignored stats)
elif "Failed" in msg or "Error" in msg:
dashboard.add_log(f"[Fail] {dashboard.format_filename(fname)}")
dashboard.update_stats("failed")
try:
encoder.process_file(file_path, category, lock_dir, log_dir, encoders, category_root, worker_id, status_callback)
except Exception as e:
dashboard.add_log(f"Error in worker {worker_id}: {str(e)[:30]}")
dashboard.update_stats("failed")
finally:
dashboard.update_worker(worker_id, "Idle", "Waiting", "dim")
# --- Monitor ---
if HAS_WATCHDOG:
class WatcherHandler(FileSystemEventHandler):
def __init__(self, queue):
self.queue = queue
def on_created(self, event):
if not event.is_directory and event.src_path.lower().endswith(('.mkv', '.mp4')):
self.queue.put(Path(event.src_path))
def on_moved(self, event):
if not event.is_directory and event.dest_path.lower().endswith(('.mkv', '.mp4')):
self.queue.put(Path(event.dest_path))
else:
# Dummy class when watchdog not available
class WatcherHandler:
def __init__(self, queue):
self.queue = queue
# --- Main ---
def main():
parser = argparse.ArgumentParser()
parser.add_argument("--tv-dir", default=common.DEFAULT_CONFIG["tv_dir"])
parser.add_argument("--content-dir", default=common.DEFAULT_CONFIG["content_dir"])
parser.add_argument("--jobs", type=int, default=4)
parser.add_argument("--monitor", action="store_true")
parser.add_argument("--cpu-only", action="store_true", help="Force software encoding")
parser.add_argument("--temp-dir", help="Override local temp directory")
parser.add_argument("--skip-until", help="Skip files until this substring is found in filename")
parser.add_argument("--av1-encoder", choices=["hw", "sw", "off"], default="hw", help="AV1 encoder: hw (hardware), sw (software), off (disable)")
parser.add_argument("--hevc-encoder", choices=["hw", "sw", "off"], default="hw", help="HEVC encoder: hw (hardware), sw (software), off (disable)")
args = parser.parse_args()
# Setup
lock_dir, log_dir = common.get_base_paths(args)
encoder.TEMP_DIR = common.get_temp_dir(args)
# Detect HW
encoders = common.detect_hardware_encoder(args)
av1_enc, hevc_enc, hw_type = encoders
# UI
dashboard = Dashboard(args.jobs, log_dir=log_dir)
dashboard.add_log(f"Locks: {lock_dir}")
dashboard.add_log(f"Logs: {log_dir}")
dashboard.add_log(f"Hardware: {hw_type.upper()}")
dashboard.add_log(f"AV1: {av1_enc or 'None'} | HEVC: {hevc_enc or 'None'}")
# Work Queue
work_queue = queue.Queue()
# Background Scanner
def background_scanner():
time.sleep(2) # Let UI start
dashboard.add_log("Starting background scan...")
# Skip-until logic
skip_until = args.skip_until
skipping = bool(skip_until) # True until we find the match
skipped_count = 0
if skip_until:
dashboard.add_log(f"Skip mode: Looking for '{skip_until}'")
def should_skip(filepath):
nonlocal skipping, skipped_count
if not skipping:
return False
# Check if skip_until substring is in the filepath
if skip_until.lower() in str(filepath).lower():
skipping = False # Found it, stop skipping
dashboard.add_log(f"Match found: {Path(filepath).name}")
dashboard.add_log(f"Resuming processing from here")
return False
skipped_count += 1
return True
# Load Cache first
cached_files = load_cache()
if cached_files:
dashboard.add_log(f"Loaded {len(cached_files)} files from cache.")
for f in cached_files:
if should_skip(f):
continue
p = Path(f)
cat = "tv_shows" if str(args.tv_dir) in str(p) else "content"
work_queue.put((p, cat))
# Real Scan
all_files = []
for d, cat in [(args.tv_dir, "tv_shows"), (args.content_dir, "content")]:
found = fast_scan(d)
for f in found:
all_files.append(f)
# Only add if NOT in cache
if str(f) not in cached_files:
if should_skip(f):
continue
work_queue.put((Path(f), cat))
if skipped_count > 0:
dashboard.add_log(f"Skipped {skipped_count} files (--skip-until)")
dashboard.add_log(f"Scan complete. Total: {len(all_files)}")
save_cache(all_files)
# Start Scanner Thread
scan_thread = threading.Thread(target=background_scanner, daemon=True)
scan_thread.start()
# Thread Pool for Workers
threads = []
def worker_loop(w_id):
# Map category to root path
category_roots = {
"tv_shows": Path(args.tv_dir),
"content": Path(args.content_dir)
}
while not encoder.shutdown_requested:
try:
item = work_queue.get(timeout=1)
file_path, category = item
category_root = category_roots.get(category)
worker_wrapper(w_id, file_path, category, category_root, lock_dir, log_dir, encoders, dashboard)
work_queue.task_done()
except queue.Empty:
# If batch mode and scan is done and queue is empty, exit
if not args.monitor and not scan_thread.is_alive() and work_queue.empty():
time.sleep(2) # Grace period
if work_queue.empty():
return # Exit thread
continue
except Exception as e:
dashboard.add_log(f"Worker {w_id} crashed: {e}")
for i in range(args.jobs):
t = threading.Thread(target=worker_loop, args=(i,), daemon=True)
t.start()
threads.append(t)
# UI Loop
try:
if HAS_RICH:
with Live(dashboard.get_renderable(), refresh_per_second=4) as live:
while not encoder.shutdown_requested:
live.update(dashboard.get_renderable())
time.sleep(0.25)
# Exit condition
if not args.monitor and not scan_thread.is_alive() and work_queue.unfinished_tasks == 0:
# Check if threads are actually dead
if all(not t.is_alive() for t in threads):
break
else:
while not encoder.shutdown_requested:
time.sleep(1)
if not args.monitor and not scan_thread.is_alive() and work_queue.empty(): break
except KeyboardInterrupt:
encoder.shutdown_requested = True
print("\nStopping...")
print("\nDone.")
if __name__ == "__main__":
main()
src/test_monitor_logic.py
-47
View File
@@ -1,47 +0,0 @@
from smart_monitor import Dashboard
import time
def test_tui_parsing():
print("Testing Dashboard Logic...")
db = Dashboard(1)
# Test 1: Encoding Progress
msg = "Encoding av1 | 45.2% | 2.3x"
db.update_worker(0, "Test File 1.mkv", msg)
status = db.worker_status[0]
print(f"Test 1 (Parsing): Progress={status['progress']} (Expected 45.2), Speed={status['speed']} (Expected 2.3x)")
assert status['progress'] == 45.2
assert status['speed'] == "2.3x"
# Test 2: Stats - Skipped
print("\nTest 2: Stats - Skipped")
# Simulate worker wrapper logic
msg = "Already AV1 (Skipping)"
if "Skipping" in msg:
db.update_stats("skipped")
print(f"Skipped Count: {db.stats['skipped']} (Expected 1)")
assert db.stats['skipped'] == 1
# Test 3: Stats - Rejected
print("\nTest 3: Stats - Rejected")
msg = "HEVC Rejected"
if "Rejected" in msg:
db.update_stats("rejected")
print(f"Rejected Count: {db.stats['rejected']} (Expected 1)")
assert db.stats['rejected'] == 1
# Test 4: Stats - Failed
print("\nTest 4: Stats - Failed")
msg = "Error: FFMPEG failed"
if "Error" in msg:
db.update_stats("failed")
print(f"Failed Count: {db.stats['failed']} (Expected 1)")
assert db.stats['failed'] == 1
print("\nAll Tests Passed!")
if __name__ == "__main__":
test_tui_parsing()
src/vmaf_common.py
-481
View File
@@ -1,481 +0,0 @@
import os
import sys
import shutil
import json
import hashlib
import platform
import subprocess
import time
from pathlib import Path
from datetime import datetime
# --- Defaults ---
DEFAULT_CONFIG = {
"tv_dir": r"Z:\tv",
"content_dir": r"Z:\content",
"target_vmaf": 93.0,
"min_savings": 12.0,
"gpu_jobs": 2,
"cpu_jobs": 1,
"av1_crf": 34,
"hevc_crf": 28,
"temp_dir_windows": r"C:\Users\bnair\Videos\encodes",
"temp_dir_macos": "/Users/bnair/Documents/encodes",
"temp_dir_linux": "~/Videos/encodes"
}
# --- Paths ---
def get_base_paths(args=None):
r"""
Determine root paths for locks and logs.
Priority:
1. Shared Network Drive (parent of tv_dir) -> Z:\.vmaf_locks
2. Local Fallback (current dir / logs)
"""
tv_dir = Path(args.tv_dir) if args and hasattr(args, 'tv_dir') else Path(DEFAULT_CONFIG["tv_dir"])
# Logic: Locks MUST be at the common root of content to be shared.
# We attempt to find the common parent if both are supplied, otherwise default to tv_dir parent.
potential_roots = []
if args:
if hasattr(args, 'tv_dir') and args.tv_dir: potential_roots.append(Path(args.tv_dir))
if hasattr(args, 'content_dir') and args.content_dir: potential_roots.append(Path(args.content_dir))
if not potential_roots:
potential_roots.append(Path(DEFAULT_CONFIG["tv_dir"]))
# Find common path
try:
if len(potential_roots) > 1:
# If they are on different drives, common_path might fail or return empty on Windows
# In that case, we fall back to the first one (TV dir parent)
common_root = os.path.commonpath(potential_roots)
shared_root = Path(common_root)
# If common root is the root drive itself (e.g. Z:\), that's fine.
# If it's a subdir (Z:\Media), also fine.
# But if it's too deep (e.g. just Z:\), we want to make sure we don't dump .vmaf_locks in root unless intended.
# Typically, we want the parent of the library folders.
# If os.path.commonpath returns Z:\Media, and dirs are Z:\Media\TV and Z:\Media\Content
# Then locks go to Z:\Media\.vmaf_locks. Perfect.
else:
shared_root = potential_roots[0].parent
except:
# Fallback for different drives
shared_root = potential_roots[0].parent
# Defaults
lock_dir = Path("locks").resolve()
log_dir = Path("logs").resolve()
# Network Logic
if shared_root.exists():
network_lock = shared_root / ".vmaf_locks"
# We prefer local logs to avoid network I/O issues during logging,
# but locks MUST be shared.
try:
network_lock.mkdir(parents=True, exist_ok=True)
lock_dir = network_lock
except Exception as e:
print(f"[Warning] Could not create network locks at {network_lock}: {e}")
print(f" Falling back to local locks: {lock_dir}")
# Ensure existence
lock_dir.mkdir(parents=True, exist_ok=True)
log_dir.mkdir(parents=True, exist_ok=True)
return lock_dir, log_dir
def get_temp_dir(args=None):
# Check args override
if args and hasattr(args, 'temp_dir') and args.temp_dir:
path = Path(args.temp_dir)
elif platform.system() == "Windows":
path = Path(DEFAULT_CONFIG["temp_dir_windows"])
elif platform.system() == "Darwin": # macOS
path = Path(DEFAULT_CONFIG["temp_dir_macos"])
else: # Linux
path = Path(DEFAULT_CONFIG["temp_dir_linux"]).expanduser()
path.mkdir(parents=True, exist_ok=True)
return path
# --- Logging ---
def log_event(log_dir, filename, data):
log_path = Path(log_dir) / filename
data["timestamp"] = datetime.now().isoformat()
try:
with open(log_path, "a", encoding="utf-8") as f:
f.write(json.dumps(data) + "\n")
except Exception as e:
print(f"[ERROR] Failed to write log: {e}")
# --- Global Processed Log ---
# Prevents re-encoding files that were already processed in previous runs
_processed_cache = None
def load_processed_log(log_dir):
"""Load set of already-processed file paths from processed.jsonl"""
global _processed_cache
if _processed_cache is not None:
return _processed_cache
_processed_cache = set()
log_path = Path(log_dir) / "processed.jsonl"
if log_path.exists():
try:
with open(log_path, "r", encoding="utf-8") as f:
for line in f:
try:
entry = json.loads(line.strip())
if "file" in entry:
_processed_cache.add(entry["file"])
except:
continue
except Exception as e:
print(f"[Warning] Could not load processed log: {e}")
return _processed_cache
def is_already_processed(log_dir, filepath):
"""Check if file was already processed in a previous run"""
processed = load_processed_log(log_dir)
return str(filepath) in processed
def mark_processed(log_dir, filepath, codec, vmaf, savings):
"""Mark a file as processed (prevents re-encoding in future runs)"""
global _processed_cache
log_path = Path(log_dir) / "processed.jsonl"
entry = {
"file": str(filepath),
"codec": codec,
"vmaf": vmaf,
"savings": savings,
"timestamp": datetime.now().isoformat()
}
try:
with open(log_path, "a", encoding="utf-8") as f:
f.write(json.dumps(entry) + "\n")
# Update cache
if _processed_cache is not None:
_processed_cache.add(str(filepath))
except Exception as e:
print(f"[Warning] Could not write to processed log: {e}")
# --- Dependencies ---
def check_dependencies(required_tools=None):
if required_tools is None:
required_tools = ["ffmpeg", "ffprobe", "ab-av1"]
missing = []
# Determine bin path relative to this file
bin_path = Path(__file__).parent.parent / "bin"
for tool in required_tools:
if tool == "ab-av1":
# Check bundled first in bin/
bundled = bin_path / "ab-av1.exe"
if bundled.exists(): continue
# Check PATH
if shutil.which("ab-av1"): continue
missing.append("ab-av1")
else:
if not shutil.which(tool):
missing.append(tool)
if missing:
print(f"[!] CRITICAL: Missing tools: {', '.join(missing)}")
print(f" Checked bundled path: {bin_path}")
sys.exit(1)
# --- Metadata ---
def get_video_info(filepath):
"""Get video info using ffprobe (Standardized)"""
try:
cmd = [
"ffprobe", "-v", "quiet",
"-print_format", "json",
"-show_format", "-show_streams",
str(filepath)
]
result = subprocess.run(cmd, capture_output=True, text=True, check=True)
data = json.loads(result.stdout)
video = next((s for s in data["streams"] if s["codec_type"] == "video"), None)
if not video: return None
size = int(data["format"].get("size", 0))
duration = float(data["format"].get("duration", 0))
# Bitrate calc
bitrate = int(data["format"].get("bitrate", 0))
if bitrate == 0 and duration > 0:
bitrate = int((size * 8) / duration)
return {
"codec": video.get("codec_name"),
"width": int(video.get("width", 0)),
"height": int(video.get("height", 0)),
"duration": duration,
"size": size,
"bitrate": bitrate,
"fps": video.get("r_frame_rate", "0/0")
}
except Exception:
return None
# --- Locks ---
def acquire_lock(lock_dir, filepath, category_root=None):
"""
Simple file-based lock with completion tracking.
Uses relative path from category root for cross-platform compatibility.
Lock lifecycle:
- Created with status="processing" when file starts
- Updated to status="completed" when file finishes successfully
- Deleted only if process is cancelled/interrupted
This way lock files act as "already processed" markers across machines.
Args:
lock_dir: Directory to store lock files
filepath: Full path to the video file
category_root: Root directory (tv_dir or content_dir) to calculate relative path
Returns lock_path if acquired, None if failed/already completed.
"""
# Calculate relative path for consistent hashing across platforms
if category_root:
try:
rel_path = Path(filepath).relative_to(category_root)
# Use forward slashes for consistency across platforms
hash_key = str(rel_path).replace(os.sep, '/')
except ValueError:
# File not under category_root, use filename
hash_key = Path(filepath).name
else:
hash_key = Path(filepath).name
fhash = hashlib.md5(hash_key.encode()).hexdigest()
lock_file = lock_dir / f"{fhash}.lock"
if lock_file.exists():
try:
# Read lock file to check status
lock_data = json.loads(lock_file.read_text())
if lock_data.get("status") == "completed":
# File already processed successfully - skip it
return None
# Check if it's a stale "processing" lock (older than 24h)
if lock_data.get("status") == "processing":
if time.time() - lock_data.get("timestamp", 0) > 86400:
# Stale lock - remove and retry
lock_file.unlink()
else:
# Fresh processing lock - file is being worked on
return None
except:
# Corrupted lock file - remove and retry
lock_file.unlink()
try:
# Create new processing lock
import socket
lock_info = {
"file": str(filepath),
"relative_path": hash_key,
"host": socket.gethostname(),
"timestamp": time.time(),
"status": "processing"
}
lock_file.write_text(json.dumps(lock_info))
return lock_file
except Exception as e:
print(f"[Warning] Failed to acquire lock for {filepath.name}: {e}")
return None
def mark_lock_completed(lock_file):
"""Mark a lock file as completed (keep it for future runs)"""
if not lock_file or not lock_file.exists():
return
try:
lock_data = json.loads(lock_file.read_text())
lock_data["status"] = "completed"
lock_data["completed_at"] = time.time()
lock_file.write_text(json.dumps(lock_data))
except Exception as e:
print(f"[Warning] Failed to mark lock as completed: {e}")
# --- Hardware Detection ---
def detect_hardware_encoder(args=None):
"""
Detects available hardware encoders via ffmpeg (Cross-Platform).
Supports --av1-encoder and --hevc-encoder flags:
hw = prefer hardware encoder (error if unavailable)
sw = force software encoder
off = disable this codec entirely
"""
# Get user preferences from args
av1_pref = getattr(args, 'av1_encoder', 'hw') if args else 'hw'
hevc_pref = getattr(args, 'hevc_encoder', 'hw') if args else 'hw'
cpu_only = getattr(args, 'cpu_only', False) if args else False
# If cpu_only, treat both as 'sw'
if cpu_only:
av1_pref = 'sw'
hevc_pref = 'sw'
try:
# Run ffmpeg -encoders
res = subprocess.run(["ffmpeg", "-hide_banner", "-encoders"], capture_output=True, text=True)
out = res.stdout
# Build available encoder maps
hw_av1_encoders = []
hw_hevc_encoders = []
hw_type = "cpu"
# 1. AMD (AMF) - Windows (Preferred)
if "av1_amf" in out: hw_av1_encoders.append(("av1_amf", "amf"))
if "hevc_amf" in out: hw_hevc_encoders.append(("hevc_amf", "amf"))
# 2. NVIDIA (NVENC) - Windows/Linux
if "av1_nvenc" in out: hw_av1_encoders.append(("av1_nvenc", "nvenc"))
if "hevc_nvenc" in out: hw_hevc_encoders.append(("hevc_nvenc", "nvenc"))
# 3. AMD (VAAPI) - Linux
if "av1_vaapi" in out: hw_av1_encoders.append(("av1_vaapi", "vaapi"))
if "hevc_vaapi" in out: hw_hevc_encoders.append(("hevc_vaapi", "vaapi"))
# 4. Intel (QSV) - Windows/Linux
if "av1_qsv" in out: hw_av1_encoders.append(("av1_qsv", "qsv"))
if "hevc_qsv" in out: hw_hevc_encoders.append(("hevc_qsv", "qsv"))
# 5. Apple Silicon (VideoToolbox) - macOS
if "av1_videotoolbox" in out: hw_av1_encoders.append(("av1_videotoolbox", "videotoolbox"))
if "hevc_videotoolbox" in out: hw_hevc_encoders.append(("hevc_videotoolbox", "videotoolbox"))
# Software encoders
has_libsvtav1 = "libsvtav1" in out
has_libx265 = "libx265" in out
# Resolve AV1 encoder
av1_enc = None
if av1_pref == 'off':
av1_enc = None
elif av1_pref == 'sw':
if has_libsvtav1:
av1_enc = "libsvtav1"
else:
print("[Warning] libsvtav1 not available, AV1 disabled")
elif av1_pref == 'hw':
if hw_av1_encoders:
av1_enc, hw_type = hw_av1_encoders[0] # First available (AMD priority)
elif has_libsvtav1:
av1_enc = "libsvtav1"
print("[Info] No AV1 HW encoder, using libsvtav1 (CPU)")
else:
print("[Warning] No AV1 encoder available")
# Resolve HEVC encoder
hevc_enc = None
if hevc_pref == 'off':
hevc_enc = None
elif hevc_pref == 'sw':
if has_libx265:
hevc_enc = "libx265"
else:
print("[Warning] libx265 not available, HEVC disabled")
elif hevc_pref == 'hw':
if hw_hevc_encoders:
hevc_enc, hevc_hw = hw_hevc_encoders[0]
# Update hw_type if we didn't get one from AV1
if hw_type == "cpu":
hw_type = hevc_hw
elif has_libx265:
hevc_enc = "libx265"
print("[Info] No HEVC HW encoder, using libx265 (CPU)")
else:
print("[Warning] No HEVC encoder available")
# Determine final hw_type label
if av1_enc and "lib" not in av1_enc:
pass # hw_type already set
elif hevc_enc and "lib" not in hevc_enc:
pass # hw_type already set from HEVC
else:
hw_type = "cpu"
return av1_enc, hevc_enc, hw_type
except Exception as e:
print(f"[Warning] HW Detection failed: {e}")
return None, None, "error"
def get_encoder_args(codec, encoder, qp):
"""Returns correct ffmpeg args for specific HW vendor"""
if not encoder: return []
# Software (CPU)
if encoder == "libsvtav1":
# CRF 0-63 (Lower is better)
return ["-c:v", "libsvtav1", "-crf", str(qp), "-preset", "6", "-g", "240"]
if encoder == "libx265":
return ["-c:v", "libx265", "-crf", str(qp), "-preset", "medium"]
# NVIDIA NVENC
if "nvenc" in encoder:
# p6 = better quality, spatial-aq for better perception
return ["-c:v", encoder, "-rc", "constqp", "-qp", str(qp), "-preset", "p6", "-spatial-aq", "1"]
# AMD AMF
if "amf" in encoder:
return ["-c:v", encoder, "-usage", "transcoding", "-rc", "cqp", "-qp_i", str(qp), "-qp_p", str(qp), "-qp_b", str(qp), "-quality", "quality"]
# Intel QSV
if "qsv" in encoder:
return ["-c:v", encoder, "-global_quality", str(qp), "-look_ahead", "1"]
# Apple VideoToolbox
if "videotoolbox" in encoder:
# Map 0-51 QP to 100-0 Quality (approx)
q = int(100 - (qp * 2))
return ["-c:v", encoder, "-q:v", str(q)]
# Linux VAAPI
if "vaapi" in encoder:
# Uses -qp normally? or -global_quality? Depends on driver.
# Often needs: -vf format=nv12,hwupload
# Safe bet for vaapi is usually CQP via -qp or -global_quality
return ["-c:v", encoder, "-qp", str(qp)]
return []
def scan_directory(directory, extensions=None):
"""Scan directory for video files"""
if extensions is None:
extensions = {".mkv", ".mp4", ".mov", ".avi", ".ts"}
files = []
for dirpath, dirnames, filenames in os.walk(directory):
# Skip processed/system files
dirnames[:] = [d for d in dirnames if not d.startswith("_") and d not in [".recycle", "@eaDir"]]
for filename in filenames:
filepath = Path(dirpath) / filename
if filepath.suffix.lower() in extensions:
if "_av1" in filepath.stem.lower() or "_hevc" in filepath.stem.lower():
continue
files.append(filepath)
return sorted(files, key=lambda x: x.stat().st_size, reverse=True)
test_run.py
-42
View File
@@ -1,42 +0,0 @@
import os
import subprocess
from pathlib import Path
# Setup
TEST_DIR = Path("test_media")
TEST_DIR.mkdir(exist_ok=True)
TEST_FILE = TEST_DIR / "test_video.mkv"
print("--- SMART ENCODER TEST ---")
# 1. Create Dummy Video (10s, 1080p, noise to ensure it's not trivial)
if not TEST_FILE.exists():
print("Generating dummy video...")
# Generate 10s video with noise
cmd = [
"ffmpeg", "-y", "-f", "lavfi", "-i", "testsrc=duration=10:size=1920x1080:rate=30",
"-f", "lavfi", "-i", "sine=frequency=1000:duration=10",
"-c:v", "libx264", "-pix_fmt", "yuv420p", "-b:v", "5M",
"-c:a", "aac",
str(TEST_FILE)
]
subprocess.run(cmd, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
print(f"Created {TEST_FILE}")
# 2. Run Encoder in Debug Mode
print("\nRunning Smart Encoder on test file...")
print("(This should show detailed ffmpeg output)")
print("="*60)
cmd = [
"python", "smart_gpu_encoder.py",
"--tv-dir", str(TEST_DIR.resolve()),
"--content-dir", str(TEST_DIR.resolve()),
"--jobs", "1",
"--debug"
]
subprocess.run(cmd)
print("\n--- TEST COMPLETE ---")
print("Check for .mkv files in C:\\Users\\bnair\\Videos\\encodes")