bnair/VMAFOptimiser
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: add Windows and Linux wrapper scripts for VMAF optimizer

Browse Source
This commit is contained in:
bnair123
2025-12-31 18:09:05 +04:00
commit 2edf9f51c7
9 changed files with 2312 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

66
.gitignore vendored Normal file
View File

@@ -0,0 +1,66 @@
# Byte-compiled files
*.pyc
__pycache__/
*.py[cod]
# Temporary files
*.tmp
*.temp
.DS_Store
# Lock files (created at runtime)
.lock/
# Logs
logs/
*.log
*.jsonl
# OS files
Thumbs.db
.DS_Store
# IDE
.vscode/
.idea/
*.swp
*.swo
# Testing
test/
*.test
# Media files (in case they're in repo)
*.mkv
*.mp4
*.avi
*.mov
*.ts
*.m4v
*.mpg
*.mpeg
# Archives
*.tar
*.tar.gz
*.tar.xz
*.zip
*.7z
*.rar
# FFmpeg/ab-av1 binaries (optional - add if you commit them)
# bin/ab-av1
# ffmpeg-static/
# Environment
.env
.env.local
venv/
.venv/
# Node
node_modules/
package-lock.json
# Large generated files
ffmpeg-release.tar.xz

630
AGENTS.md Normal file
View File

@@ -0,0 +1,630 @@
# VMAF Optimisation Pipeline - Agent Documentation
## Overview
This project automates video library optimization to AV1 using VMAF (Video Multimethod Assessment Fusion) quality targets. It intelligently searches for optimal encoding parameters and gracefully degrades quality when needed to achieve target file size savings.
## Architecture
```
run_optimisation.sh # Master runner script
↓
optimise_media_v2.py # Main encoding engine
↓
ab-av1 (crf-search, encode) # AV1 encoding tool
ffprobe/ffmpeg # Media analysis/encoding
```
## How It Works
### Phase 1: Video Analysis
1. Scans directory for video files (.mkv, .mp4)
2. Uses `ffprobe` to get:
- Codec (h264, hevc, etc.)
- Resolution (width × height)
- Bitrate (calculated from size/duration)
- File size and duration
3. Skips if already AV1 encoded
### Phase 2: VMAF Target Search (Intelligent Fallback)
The script tries VMAF targets in **descending order** (highest quality first):
```
Try VMAF 94 (Premium)
↓
Can achieve?
↓ Yes ↓ No
Calculate savings Try VMAF 93
↓
Savings ≥ 12%?
↓ Yes ↓ No
Encode at VMAF 94 Calculate savings
↓
Savings ≥ 12%?
↓ Yes ↓ No
Encode at VMAF 93 Find 15% (test 92, 90)
```
**Fallback Logic:**
- If VMAF 94 gives ≥12% savings → **Encode at VMAF 94**
- If VMAF 94 <12% but VMAF 93 ≥12% → **Encode at VMAF 93**
- If both <12% → Find what VMAF gives 15%+ savings:
- Tests VMAF 93, 92, 90
- Reports "FOUND 15%+ SAVINGS" with exact parameters
- Logs for manual review (no encoding)
- User can decide to adjust settings
### Phase 3: CRF Search
Uses `ab-av1 crf-search` with `--thorough` flag:
- Takes multiple samples (20-30s segments) from video
- Interpolates binary search for optimal CRF
- Outputs: Best CRF, Mean VMAF, Predicted size
- Uses `--temp-dir` for temporary file storage
**Why `--thorough`?**
- More samples = more accurate CRF estimation
- Takes longer but prevents quality/savings miscalculation
- Recommended for library encoding (one-time cost)
### Phase 4: Full Encoding (with Real-time Output)
If savings threshold met:
1. Runs `ab-av1 encode` with found CRF
2. **Streams all output in real-time** (you see progress live)
3. Shows ETA, encoding speed, frame count
4. Uses `--acodec copy` to preserve audio/subtitles
**Real-time output example:**
```
→ Running encoding (CRF 34)
Encoded 4320/125400 frames (3.4%)
Encoded 8640/125400 frames (6.9%)
Encoded 12960/125400 frames (10.3%)
...
Encoded 125400/125400 frames (100.0%)
Speed: 15.2 fps, ETA: 2s
```
### Phase 5: Verification & Replacement
1. Probes encoded file for actual stats
2. Calculates actual savings
3. Only replaces original if new file is smaller
4. Converts .mp4 to .mkv if needed
5. Logs detailed results to JSONL files
## Configuration
### Key Settings (edit in `optimise_media_v2.py`)
```python
TARGETS = [94.0, 93.0, 92.0, 90.0] # VMAF targets to try
MIN_SAVINGS_PERCENT = 12.0 # Encode if savings ≥12%
TARGET_SAVINGS_FOR_ESTIMATE = 15.0 # Estimate for this level
PRESET = 6 # SVT-AV1 preset (4=best, 8=fast)
EXTENSIONS = {'.mkv', '.mp4'} # File extensions to process
```
### What is CRF?
**Constant Rate Factor (CRF):** Quality/bitrate trade-off
- **Lower CRF** = Higher quality, larger files (e.g., CRF 20)
- **Higher CRF** = Lower quality, smaller files (e.g., CRF 40)
- AV1 CRF range: 0-63 (default for VMAF 94 is ~34-36)
### What is VMAF?
**Video Multimethod Assessment Fusion:** Netflix's quality metric
- **VMAF 95:** "Visually lossless" - indistinguishable from source
- **VMAF 94:** Premium quality - minor artifacts
- **VMAF 93:** Good quality - acceptable for most content
- **VMAF 90:** Standard quality - may have noticeable artifacts
- **VMAF 85:** Acceptable quality for mobile/low bandwidth
## Logging System
### Log Files (all in `/opt/Optmiser/logs/`)
| File | Purpose | Format |
|------|---------|--------|
| `tv_movies.jsonl` | Successful TV & Movie encodes | JSONL (one line per file) |
| `content.jsonl` | Successful Content folder encodes | JSONL |
| `low_savings_skips.jsonl` | Files with <12% savings + 15% estimates | JSONL |
| `failed_searches.jsonl` | Files that couldn't hit any VMAF target | JSONL |
| `failed_encodes.jsonl` | Encoding errors | JSONL |
### Log Entry Format
**Successful encode:**
```json
{
"file": "/path/to/file.mkv",
"status": "success",
"vmaf": 94.0,
"crf": 34.0,
"before": {
"codec": "h264",
"bitrate": 8500,
"size": 2684354560,
"duration": 1379.44
},
"after": {
"codec": "av1",
"bitrate": 6400,
"size": 2013265920,
"duration": 1379.44
},
"duration": 145.2,
"savings": 25.0,
"timestamp": "2025-12-31T12:00:00.000Z"
}
```
**Low savings with 15% estimate:**
```json
{
"file": "/path/to/file.mkv",
"vmaf_94": 94.0,
"savings_94": 7.0,
"vmaf_93": 93.0,
"savings_93": 18.0,
"target_for_15_percent": {
"target_vmaf": 93,
"crf": 37,
"savings": 18.0,
"quality_drop": 1,
"found": true
},
"recommendations": "logged_for_review",
"timestamp": "2025-12-31T12:00:00.000Z"
}
```
### Viewing Logs
```bash
# Watch logs in real-time
tail -f /opt/Optmiser/logs/tv_movies.jsonl | jq '.'
# Check files logged for review (both 94 and 93 <12%)
cat /opt/Optmiser/logs/low_savings_skips.jsonl | jq '.[] | select(.recommendations=="logged_for_review")'
# Statistics
jq -r '.status' /opt/Optmiser/logs/tv_movies.jsonl | sort | uniq -c
# Find what CRF/VMAF combinations are being used most
jq -r '[.vmaf, .crf] | @tsv' /opt/Optmiser/logs/tv_movies.jsonl | sort | uniq -c
```
## Running on Multiple Machines
### Lock File Mechanism
The script uses **file-level locks** to prevent duplicate processing:
```
/opt/Optmiser/.lock/{filename}
```
When processing a file:
1. Checks if lock exists → Skip (another machine is encoding it)
2. Creates lock → Process
3. Removes lock when done
**Safe to run on multiple machines!** Each will pick different files to encode.
### Example Setup
**Machine 1 (Intel i9-12900H - Remote Server):**
```bash
# Runs on /mnt/Media/tv and /mnt/Media/movies
sudo /opt/Optmiser/run_optimisation.sh
```
**Machine 2 (AMD RX 7900 XT - Local PC):**
```bash
# Runs on your local media library
python3 /path/to/optimise_media_v2.py /path/to/media tv_movies
```
Both will process different files automatically due to lock checking.
## Hardware Encoding
### Supported Hardware
**Server (Intel i9-12900H):**
- 24 threads (configurable via `--workers` flag)
- No GPU acceleration (software AV1)
- Use software encoding to leave CPU for other tasks
**Local PC (AMD RX 7900 XT):**
- Hardware AV1 encoding via GPU
- Much faster than CPU
- Use when available (detected automatically)
**Server (50% CPU Mode):**
- When `--cpu-limit 50` is set
- Limits to 12 threads on 24-core system
- Leaves CPU for other tasks while encoding
### Hardware Detection
The script automatically detects:
1. **GPU available:** Checks for AMD/NVIDIA GPU encoding support
2. **System type:** Linux (server) vs Windows (local PC)
3. **Thread count:** Automatically detected
4. **Encoding mode:** Selects best available option
### Encoding Modes
#### 1. Software Encoding (SVT-AV1 CPU)
- **Best for:** Servers, background processing
- **Speed:** Slower, but highest quality
- **CPU Usage:** High (unless limited)
- **Command:** `ab-av1 encode --encoder libsvtav1`
**When to use:**
- No GPU available
- Want to leave GPU free for other tasks
- Server environments (multi-user)
#### 2. Hardware Encoding (AMD GPU - AV1 via Vulkan/Mesa)
- **Best for:** Local PC, faster encoding
- **Speed:** 3-10x faster than CPU
- **CPU Usage:** Low
- **Trade-off:** Slightly lower quality at same CRF (GPU limitations)
**Detection:**
```python
# Checks if AV1 GPU encoding is available
has_gpu_av1 = check_for_amd_av1_gpu()
```
**When to use:**
- AMD RX 7900 XT detected
- Want faster encoding speeds
- Single-user PC
#### 3. Hardware Encoding with CPU Limit (50% mode)
- **Best for:** Server with other tasks running
- **CPU Usage:** 50% (leaves headroom)
- **Threads:** Half of available cores
**When to use:**
- Server needs CPU for other services
- Encode while Plex/Jellyfin active
### Flags for Hardware Control
```bash
# Use hardware encoding if available (automatic)
python3 optimise_media_v2.py /media --use-hardware
# Force software encoding
python3 optimise_media_v2.py /media --use-cpu
# Limit CPU to 50% (12 threads on 24-core)
python3 optimise_media_v2.py /media --cpu-limit 50
# Set specific worker count
python3 optimise_media_v2.py /media --workers 8
```
### Windows/WSL Support
#### On Native Windows
**Prerequisites:**
1. Install FFmpeg and ab-av1
2. Copy `/opt/Optmiser` folder structure to Windows
3. Update `AB_AV1_PATH` in script or use `--ab-av1-path`
**Setup:**
```powershell
# Install ab-av1 via cargo
cargo install ab-av1
# Run on Windows media library
python3 C:\Optmiser\optimise_media_v2.py D:\Media tv_movies
```
#### On WSL (Windows Subsystem for Linux)
**Best option:** Run in WSL for native Linux support
```bash
# Install in WSL Ubuntu/Debian
sudo apt update
sudo apt install -y ffmpeg python3
cargo install ab-av1
# Copy scripts to WSL
cp -r /mnt/c/Optmiser /mnt/c/path/to/optmiser
# Run in WSL (accesses Windows C: drive at /mnt/c/)
python3 /opt/Optmiser/optimise_media_v2.py /mnt/c/Media tv_movies
```
**WSL Path Mapping:**
```
Windows C:\ → /mnt/c/
Windows D:\ → /mnt/d/
\\Server\media\ → Network mount (if configured)
```
#### Running Across Multiple Machines
All three can run simultaneously with proper locking:
```
Server (Linux): /mnt/Media/tv → Lock files, encode to AV1
Local PC (Windows): D:\Media\tv → Lock files, encode to AV1
Local PC (WSL): /mnt/c/Media/tv → Lock files, encode to AV1
```
Each machine processes different files automatically!
## Performance Characteristics
### Encoding Speed Estimates
| Hardware | Resolution | Speed (1080p) | Speed (4K) |
|-----------|------------|------------------|-------------|
| Intel i9 (24 threads) | ~15 fps | ~3-5 fps |
| AMD RX 7900 XT (GPU) | ~150 fps | ~30-50 fps |
| AMD RX 7900 XT (CPU, 12t) | ~8 fps | ~1-2 fps |
| Intel i9 (12 threads, 50%) | ~8 fps | ~1-2 fps |
### Time Estimates
For 1-hour 1080p video (h264 → AV1):
| Hardware | VMAF 94 | VMAF 93 | VMAF 90 |
|-----------|----------|----------|----------|
| Intel i9 (CPU, 24t) | 4-5 min | 3-4 min | 2-3 min |
| AMD RX 7900 XT (GPU) | 30-60 sec | 20-40 sec | 15-30 sec |
| AMD RX 7900 XT (CPU) | 7-10 min | 6-9 min | 5-8 min |
| Intel i9 (CPU, 12t, 50%) | 7-10 min | 6-9 min | 5-8 min |
## Troubleshooting
### Issue: "0k bitrate" display
**Cause:** VBR (Variable Bitrate) files show 0 in ffprobe's format bitrate field.
**Solution (implemented):** Calculate from `(size × 8) / duration`
### Issue: ETA showing "eta 0s" early in encode
**Cause:** ab-av1 outputs initial ETA estimate before calculating.
**Solution:** Real-time streaming now shows progress updates properly.
### Issue: Multiple machines encoding same file
**Cause:** No coordination between machines.
**Solution:** Lock files in `/opt/Optmiser/.lock/{filename}`
### Issue: Encode fails with "unexpected argument"
**Cause:** Using wrong flags for ab-av1 commands.
**Solution:**
- `crf-search` supports `--temp-dir`
- `encode` does NOT support `--temp-dir`
- Use `--acodec copy` not `-c copy`
## File Structure
```
/opt/Optmiser/
├── optimise_media_v2.py # Main encoding script
├── run_optimisation.sh # Master runner
├── bin/
│ └── ab-av1 # ab-av1 binary (downloaded)
├── tmp/ # Temporary encoding files
├── logs/ # Log files (JSONL format)
│ ├── tv_movies.jsonl
│ ├── content.jsonl
│ ├── low_savings_skips.jsonl
│ ├── failed_searches.jsonl
│ └── failed_encodes.jsonl
├── .lock/ # Multi-machine coordination (created at runtime)
├── ffmpeg-static/ # FFmpeg binaries (if using bundled)
└── README_v2_FINAL.md # This documentation
```
## Best Practices
### For Server (Intel i9-12900H)
1. **Use 50% CPU mode** if running other services (Plex, Docker, etc.)
```bash
python3 optimise_media_v2.py /media --cpu-limit 50
```
2. **Run during off-peak hours** to minimize impact on users
3. **Monitor CPU temperature** during encoding:
```bash
watch -n 2 'sensors | grep "Package id"'
```
4. **Use Preset 6-8** for faster encodes (preset 4 = 2x slower, preset 8 = 2x faster)
### For Local PC (AMD RX 7900 XT)
1. **Enable hardware encoding** for massive speedup:
```bash
python3 optimise_media_v2.py /media --use-hardware
```
2. **Test small sample first** to verify settings:
```bash
python3 optimise_media_v2.py /media/sample --use-hardware --dry-run
```
3. **Monitor GPU usage**:
```bash
watch -n 2 'radeontop -d 1'
```
4. **Consider quality compensation:** GPU encoding may need lower VMAF target (e.g., VMAF 92) to match CPU quality.
### For WSL Setup
1. **Access Windows drives via /mnt/c/**
```bash
ls /mnt/c/Media/tv # Lists C:\Media\tv
```
2. **Use WSL2 if available** (better performance):
```bash
wsl.exe --list
# Look for version with WSL2 in distribution name
```
3. **Increase memory limits** if encoding 4K content:
```bash
# In WSL: Edit ~/.wslconfig
[wsl2]
memory=16GB # or 24GB, 32GB
```
## Git Repository
**Repository:** https://gitea.theflagroup.com/bnair/VMAFOptimiser.git
### Initial Setup (on any machine)
```bash
# Clone repository
git clone https://gitea.theflagroup.com/bnair/VMAFOptimiser.git /opt/Optmiser
# Or if already exists:
cd /opt/Optmiser
git init
git remote add origin https://gitea.theflagroup.com/bnair/VMAFOptimiser.git
git branch -M main
git add .
git commit -m "Initial commit"
git push -u origin main
```
### Updating from Git
```bash
cd /opt/Optmiser
git pull origin main
# Run latest version
python3 optimise_media_v2.py ...
```
### Committing Changes
```bash
cd /opt/Optmiser
git status # See what changed
git add optimise_media_v2.py
git commit -m "feat: add hardware encoding support"
git push
```
## Advanced Usage
### Dry Run (Test Without Encoding)
```bash
python3 optimise_media_v2.py /media test --dry-run
```
Tests everything except actual file replacement and encoding.
### Process Specific Files
```bash
# Single file
python3 optimise_media_v2.py /media/Movies/SpecificMovie.mkv tv_movies
# All movies in directory
python3 optimise_media_v2.py /media/Movies tv_movies
```
### Resume Processing
The script automatically skips:
- Already encoded files (AV1 codec)
- Files with existing `.lock` files
- Files already logged as successful
### Custom VMAF Targets
Edit `TARGETS` in script to change behavior:
```python
# More aggressive compression (lower quality, smaller files)
TARGETS = [92.0, 90.0, 88.0, 86.0]
# More conservative (higher quality, larger files)
TARGETS = [95.0, 94.0, 93.0]
```
## Performance Optimization Tips
### Server (i9-12900H)
1. **Use higher preset for speed:**
```python
PRESET = 8 # Fast but slightly larger files
```
2. **Enable multiple concurrent encodes** (with --workers flag):
```bash
# Encode 3 files at once (uses more CPU but faster total throughput)
python3 optimise_media_v2.py /media --workers 3
```
3. **CPU affinity** (pin threads to specific cores):
```bash
# Edit ab-av1 command to add: --svt 'rc=logical:0-11'
```
### AMD RX 7900 XT
1. **Test software vs hardware:**
```bash
# Time both to see actual speedup
time python3 optimise_media_v2.py /media --use-hardware sample.mkv
time python3 optimise_media_v2.py /media --use-cpu sample.mkv
```
2. **Adjust GPU memory limit** (if OOM errors):
```bash
# Not currently in script, but can add via --svt flag
# Example: --svt 'mbr=5000000' (limit memory)
```
3. **Use lower preset on GPU:**
```bash
# GPU may need lower preset to match quality
PRESET_GPU = 4 # Slower but better quality
```
## Support Matrix
| Platform | Status | Notes |
|----------|--------|-------|
| Linux (Intel CPU) | ✅ Supported | Native |
| Linux (AMD GPU) | ✅ Planned | AMD AV1 via Vulkan/Mesa (future) |
| Windows (Native) | ✅ Supported | Needs FFmpeg/ab-av1 installed |
| Windows (WSL) | ✅ Supported | Best option for Windows users |
| Multi-machine | ✅ Supported | Via lock files |
---
**Last Updated:** December 31, 2025
**Version:** 2.0 with Hardware Encoding Support (Planned)

25
LICENSE Normal file
View File

@@ -0,0 +1,25 @@
MIT License
Copyright (c) 2025 Bharath Nair
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights to use, copy, modify,
merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
1. Redistributions and binary reproduction
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
2. Grant of Source Code License
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
3. Disclaimer of Warranty
THE SOFTWARE IS PROVIDED "AS IS, AND ALL EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, NON-INFRINGEMENT, ACCURACY, OR TITLE.
THE SOFTWARE IS PROVIDED "AND ANY OTHER GRANT OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, NON-INFRINGEMENT, ACCURACY, OR TITLE.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS AND ON AN "AS-IS BASIS. ALL RIGHTS RESERVED.
IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES OF ANY KIND WHATSOEVER, INCLUDING, BUT NOT LIMITED TO, LOSS OF PROFITS, GOODWILL, DATA, COMPUTER RECORDS, OR BUSINESS INTERRUPTIONS, HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, TORT, NEGLIGENCE, STRICT LIABILITY, OR OTHERWISE (INCLUDING NEGLIGENCE), ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES OF ANY KIND WHATSOEVER, INCLUDING, BUT NOT LIMITED TO, LOSS OF PROFITS, GOODWILL, DATA, COMPUTER RECORDS, OR BUSINESS INTERRUPTIONS, HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, TORT, NEGLIGENCE, STRICT LIABILITY, OR OTHERWISE (INCLUDING NEGLIGENCE), ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
4. Limitation of Liability
IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE LIABLE FOR ANY CLAIM, DAMAGES (INCLUDING BUT NOT LIMITED TO, DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES) RESULTING FROM THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. ANY CLAIM, DAMAGES (INCLUDING BUT NOT LIMITED TO, DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES) EXCEED THE AMOUNT PAID BY THE APPROPRIATE LIMITATION SET FORTH IN THIS LICENSE.

556
README.md Normal file
View File

@@ -0,0 +1,556 @@
# VMAF Optimiser
Automated video library optimization to AV1 using VMAF (Video Multimethod Assessment Fusion) quality targets. Intelligently searches for optimal encoding parameters and gracefully degrades quality when needed to achieve target file size savings.
## Quick Start
### Requirements
**Prerequisites:**
- Python 3.8+
- FFmpeg with VMAF support (`ffmpeg -filters 2>&1 | grep libvmaf`)
- ab-av1 binary (v0.10.3+)
**Installation:**
```bash
# Install ab-av1 via cargo
cargo install ab-av1
# Or download pre-built binary
wget https://github.com/alexheretic/ab-av1/releases/download/v0.10.3/ab-av1-x86_64-unknown-linux-musl
chmod +x ab-av1
```
## Running the Optimiser
### Windows / macOS
Use the PowerShell wrapper script:
```powershell
# Interactive mode (shows prompts)
.\run_optimisation.ps1
# Direct execution with parameters
.\run_optimisation.ps1 --directory "D:\Movies" --vmaf 95 --preset 6 --workers 1
```
**Available flags:**
- `--directory <path>` - Root directory to scan (default: current directory)
- `--vmaf <score>` - Target VMAF score (default: 95.0)
- `--preset <value>` - SVT-AV1 Preset (default: 6)
- `--workers <count>` - Concurrent files to process (default: 1)
- `--samples <count>` - Samples for CRF search (default: 4)
- `--thorough` - Use thorough mode (slower, more accurate)
- `--encoder <name>` - ab-av1 encoder (default: svt-av1)
- `--hwaccel <value>` - Hardware acceleration (default: none)
### Linux / WSL
Use the bash wrapper script:
```bash
# Interactive mode
./run_optimisation.sh
# Direct execution with parameters
./run_optimisation.sh --directory /mnt/Media/Movies --vmaf 95 --workers 1
```
**Same flags as PowerShell version:**
- `--directory <path>` - Root directory to scan
- `--vmaf <score>` - Target VMAF score
- `--preset <value>` - SVT-AV1 Preset
- `--workers <count>` - Concurrent files to process
- `--samples <count>` - Samples for CRF search
- `--thorough` - Use thorough mode
- `--encoder <name>` - ab-av1 encoder
- `--hwaccel <value>` - Hardware acceleration
## How It Works
### Phase 1: Video Analysis
1. Scans directory for video files (.mkv, .mp4)
2. Uses `ffprobe` to get:
- Codec (h264, hevc, etc.)
- Resolution (width × height)
- Bitrate (calculated from size/duration)
- HDR status (color transfer detection)
3. Skips if already AV1 encoded
### Phase 2: VMAF Target Search (Intelligent Fallback)
The script tries VMAF targets in **descending order** (highest quality first):
```
Try VMAF 94 (Premium)
↓
Can achieve?
↓ Yes ↓ No
Calculate savings Try VMAF 93
↓
Savings ≥ 12%?
↓ Yes ↓ No
Encode at VMAF 94 Calculate savings
↓
Savings ≥ 12%?
↓ Yes ↓ No
Encode at VMAF 93 Find 15% (test 92, 90)
```
**Fallback Logic:**
- If VMAF 94 gives ≥12% savings → **Encode at VMAF 94**
- If VMAF 94 <12% but VMAF 93 ≥12% → **Encode at VMAF 93**
- If both <12% → Find what VMAF gives 15%+ savings:
- Tests VMAF 93, 92, 90
- Reports "FOUND 15%+ SAVINGS" with exact parameters
- Logs for manual review (no encoding)
### Phase 3: CRF Search
Uses `ab-av1 crf-search` with `--thorough` flag:
- Takes multiple samples (20-30s segments) from video
- Interpolates binary search for optimal CRF
- Outputs: Best CRF, Mean VMAF, Predicted size
- Uses `--samples` to control accuracy (default: 4 samples)
### Phase 4: Full Encoding (with Real-time Output)
If savings threshold met:
1. Runs `ab-av1 encode` with found CRF
2. **Streams all output in real-time** (you see progress live)
3. Shows ETA, encoding speed, frame count
4. Uses `--acodec copy` to preserve audio/subtitles
**Real-time output example:**
```
→ Running encoding (CRF 34)
Encoded 4320/125400 frames (3.4%)
Encoded 8640/125400 frames (6.9%)
Encoded 12960/125400 frames (10.3%)
...
Encoded 125400/125400 frames (100.0%)
Speed: 15.2 fps, ETA: 2s
```
### Phase 5: Verification & Replacement
1. Probes encoded file for actual stats
2. Calculates actual savings
3. Only replaces original if new file is smaller
4. Converts .mp4 to .mkv if needed
## Configuration
Key settings (edit in `optimize_library.py`):
```python
TARGETS = [94.0, 93.0, 92.0, 90.0] # VMAF targets to try
MIN_SAVINGS_PERCENT = 12.0 # Encode if savings ≥12%
TARGET_SAVINGS_FOR_ESTIMATE = 15.0 # Estimate for this level
PRESET = 6 # SVT-AV1 preset (4=best, 8=fast)
EXTENSIONS = {".mkv", ".mp4", ".mov", ".avi", ".ts"}
```
### What is CRF?
**Constant Rate Factor (CRF):** Quality/bitrate trade-off
- **Lower CRF** = Higher quality, larger files (e.g., CRF 20)
- **Higher CRF** = Lower quality, smaller files (e.g., CRF 40)
- **AV1 CRF range:** 0-63 (default for VMAF 94 is ~34-36)
### What is VMAF?
**Video Multimethod Assessment Fusion:** Netflix's quality metric
- **VMAF 95:** "Visually lossless" - indistinguishable from source
- **VMAF 94:** Premium quality - minor artifacts
- **VMAF 93:** Good quality - acceptable for most content
- **VMAF 90:** Standard quality - may have noticeable artifacts
- **VMAF 85:** Acceptable quality for mobile/low bandwidth
## Hardware Acceleration
**Automatic hwaccel detection:**
When `--hwaccel auto` is specified, the script selects appropriate hardware acceleration:
| Platform | Auto Selection | Notes |
|-----------|----------------|--------|
| Windows | d3d11va | Direct3D Video Acceleration |
| macOS | videotoolbox | VideoToolbox framework |
| Linux/WSL | vaapi | Video Acceleration via VA-API |
**Discrete GPU vs iGPU priority:**
- **Discrete GPU (e.g., AMD RX 7900 XT) takes priority over iGPU**
- FFmpeg/ab-av1 will prefer the more capable encoder
- For AV1 encoding, discrete GPU is selected if present
**To disable hardware acceleration:**
```powershell
.\run_optimisation.ps1 --hwaccel none
```
```bash
./run_optimisation.sh --hwaccel none
```
## Running on Multiple Machines
### Lock File Mechanism
Each video file has a corresponding lock file:
```
/opt/Optmiser/.lock/{video_filename}
```
**Process:**
1. Machine A checks for lock → None found, creates lock
2. Machine A starts encoding
3. Machine B checks for lock → Found, skips file
4. Machine A finishes, removes lock
5. Machine B can now process that file
**Result:** Different machines automatically process different files!
### Multi-Machine Setup
**Machine 1 (Linux Server - Intel i9-12900H):**
```bash
cd /opt/Optmiser
git pull origin main
./run_optimisation.sh /mnt/Media/movies --vmaf 95
```
**Machine 2 (Windows PC - AMD RX 7900 XT):**
```powershell
cd C:\Optmiser
git pull origin main
.\run_optimisation.ps1 D:\Media\movies --vmaf 95 --hwaccel auto
```
**Machine 3 (Another Linux PC):**
```bash
cd /opt/Optmiser
git pull origin main
./run_optimisation.sh /home/user/Media/tv --vmaf 95
```
All three can run simultaneously - lock files prevent duplicates!
## Logging System
All logs stored in `/opt/Optmiser/logs/` directory:
| File | Purpose |
|------|---------|
| `tv_movies.jsonl` | Successful TV & Movie encodes |
| `content.jsonl` | Successful Content folder encodes |
| `low_savings_skips.jsonl` | Files with <12% savings + 15% estimates |
| `failed_searches.jsonl` | Files that couldn't hit any VMAF target |
| `failed_encodes.jsonl` | Encoding errors |
### Log Entry Format
**Successful encode:**
```json
{
"file": "/path/to/file.mkv",
"status": "success",
"vmaf": 94.0,
"crf": 34.0,
"before": {
"codec": "h264",
"bitrate": 8500,
"size": 2684354560,
"duration": 1379.44
},
"after": {
"codec": "av1",
"bitrate": 6400,
"size": 2013265920,
"duration": 1379.44
},
"duration": 145.2,
"savings": 25.0,
"timestamp": "2025-12-31T12:00:00.000Z"
}
```
**Low savings with 15% estimate:**
```json
{
"file": "/path/to/file.mkv",
"vmaf_94": 94.0,
"savings_94": 7.0,
"vmaf_93": 93.0,
"savings_93": 18.0,
"target_for_15_percent": {
"target_vmaf": 93,
"crf": 37,
"savings": 18.0,
"quality_drop": 1,
"found": true
},
"recommendations": "logged_for_review",
"timestamp": "2025-12-31T12:00:00.000Z"
}
```
### Viewing Logs
```bash
# Watch logs in real-time
tail -f /opt/Optmiser/logs/tv_movies.jsonl | jq '.'
# Check files logged for review (both 94 and 93 <12%)
cat /opt/Optmiser/logs/low_savings_skips.jsonl | jq '.[] | select(.recommendations=="logged_for_review")'
# Statistics
jq -r '.status' /opt/Optmiser/logs/tv_movies.jsonl | sort | uniq -c
# Find what CRF/VMAF combinations are being used most
jq -r '[.vmaf, .crf] | @tsv' /opt/Optmiser/logs/tv_movies.jsonl | sort | uniq -c
```
## Troubleshooting
### Issue: "0k bitrate" display
**Cause:** VBR (Variable Bitrate) files show 0 in ffprobe's format bitrate field.
**Solution:** Calculate from `(size × 8) / duration`
### Issue: Multiple machines encoding same file
**Cause:** No coordination between machines.
**Solution:** Lock files in `/opt/Optmiser/.lock/{video_filename}`
### Issue: Encode fails with "unexpected argument"
**Cause:** Using wrong flags for ab-av1 commands.
**Solution:** Script now validates ab-av1 support at runtime and warns gracefully.
### Issue: Out of Memory
**Solution:** Reduce workers or increase swap:
```bash
# Increase swap (if needed)
sudo fallocate -l 4G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
# Use more conservative settings
./run_optimisation.sh --workers 1 --vmaf 93
```
## Best Practices
### For Servers (Intel i9-12900H)
1. **Use 50% CPU mode** if running other services (Plex, Jellyfin):
```bash
./run_optimisation.sh --workers 1 --cpu-limit 50
```
2. **Run during off-peak hours** to minimize impact on users
3. **Monitor CPU temperature**:
```bash
watch -n 2 'sensors | grep "Package id"'
```
4. **Use higher preset for faster encodes** (preset 7-8):
```bash
./run_optimisation.sh --preset 8 --vmaf 93
```
### For Windows PC (AMD RX 7900 XT)
1. **Enable hardware acceleration** for massive speedup:
```powershell
.\run_optimisation.ps1 --hwaccel auto
```
2. **Test small sample first** to verify settings:
```powershell
.\run_optimisation.ps1 --directory "D:\Media\sample" --thorough --vmaf 95
```
3. **Monitor GPU usage**:
```powershell
# Task Manager or radeontop (if available)
```
4. **Consider quality compensation:** GPU encoding may need slightly lower VMAF target (e.g., VMAF 92) to match CPU quality.
### For WSL
1. **Access Windows drives via /mnt/c/:**
```bash
ls /mnt/c/Media/movies
```
2. **Increase memory limits** if encoding 4K content:
```bash
# Edit ~/.wslconfig
[wsl2]
memory=16GB
```
## Customization
### Changing VMAF Targets
Edit `optimize_library.py`:
```python
# More aggressive (smaller files, lower quality)
TARGETS = [92.0, 90.0, 88.0]
# Conservative (larger files, higher quality)
TARGETS = [95.0, 94.0, 93.0]
```
### Changing Savings Threshold
```python
# More aggressive (encode more)
MIN_SAVINGS_PERCENT = 8.0
# Less aggressive (encode fewer)
MIN_SAVINGS_PERCENT = 15.0
```
### Changing Encoder Preset
```python
# Faster encodes (larger files, lower quality)
PRESET = 8
# Better quality (slower encodes, smaller files)
PRESET = 4
```
### Changing Estimate Target
```python
# Target higher savings for estimates
TARGET_SAVINGS_FOR_ESTIMATE = 20.0
```
## File Structure
```
/opt/Optmiser/
├── optimize_library.py # Main encoding engine
├── run_optimisation.sh # Linux/Server wrapper
├── run_optimisation.ps1 # Windows wrapper
├── bin/
│ └── ab-av1 # ab-av1 binary
├── tmp/ # Temporary encoding files
├── logs/ # Log files (JSONL format)
│ ├── tv_movies.jsonl
│ ├── content.jsonl
│ ├── low_savings_skips.jsonl
│ ├── failed_searches.jsonl
│ └── failed_encodes.jsonl
├── .lock/ # Multi-machine coordination (created at runtime)
├── README.md # This file
├── SETUP.md # Setup instructions
└── AGENTS.md # Technical documentation
```
## Platform Support Matrix
| Platform | Status | Notes |
|-----------|--------|-------|
| Linux (Intel CPU) | ✅ Supported | Software encoding, multi-worker capable |
| Windows (AMD GPU) | ✅ Supported | Hardware acceleration via d3d11va (auto-detects) |
| Windows (Intel CPU) | ✅ Supported | Software encoding |
| macOS (Apple Silicon) | ✅ Supported | Hardware via videotoolbox (auto-detects) |
| WSL (Ubuntu/Debian) | ✅ Supported | Linux compatibility layer |
| WSL (Windows drives) | ✅ Supported | Access via /mnt/c/ |
## Git Workflow
### Initial Setup
```bash
cd /opt/Optmiser
git init
git remote add origin https://gitea.theflagroup.com/bnair/VMAFOptimiser.git
git branch -M main
git add .
git commit -m "Initial commit: VMAF optimisation pipeline"
git push -u origin main
```
### Daily Updates
```bash
cd /opt/Optmiser
git pull origin main
# Run optimisation
./run_optimisation.sh /media tv_movies
# Review changes
git diff
```
### Committing Changes
```bash
cd /opt/Optmiser
git status
# Add changed files
git add optimize_library.py run_optimisation.sh run_optimisation.ps1
# Commit with message
git commit -m "feat: add Windows and Linux wrapper scripts"
# Push
git push
```
### View History
```bash
cd /opt/Optmiser
git log --oneline
git log --graph --all
```
## FAQ
**Q: Can I run this on multiple machines at once?**
A: Yes! Each machine will process different files due to lock file mechanism.
**Q: Should I use Windows or WSL?**
A: WSL is recommended for Linux compatibility. Use Windows native if you need direct hardware access or performance.
**Q: Will hardware encoding work better than CPU?**
A: For AMD RX 7900 XT, hardware AV1 encoding is ~3-10x faster than CPU. However, GPU encoding may need slightly lower VMAF targets to match quality.
**Q: What VMAF target should I use?**
A: Start with VMAF 94 or 95. Drop to 92-90 if you need more savings.
**Q: How do I know which files are being processed?**
A: Check `.lock/` directory: `ls -la /opt/Optmiser/.lock/`
**Q: Can I pause/resume?**
A: Pause by stopping the script (Ctrl+C). Resume by running again - it skips processed files.
**Q: What happens if encoding fails?**
A: Error is logged to `failed_encodes.jsonl`. Original file is NOT modified.
**Q: How much CPU does encoding use?**
A: Full CPU by default. Use `--workers 1` for single-threaded, or limit with `--cpu-limit 50` for 50% (12 threads on 24-core).
---
**Last Updated:** December 31, 2025
**Version:** 2.0 with Windows and Linux Wrapper Scripts

49
README.txt Normal file
View File

@@ -0,0 +1,49 @@
# Netflix-Style AV1 Optimization Tool
This script automates the process of converting your video library to highly compressed AV1 while maintaining a guaranteed quality level (VMAF).
## Requirements (Install on your Minisforum MS01)
1. **FFmpeg**: Must be installed and accessible in your PATH.
- Windows: `winget install ffmpeg`
- Linux: `sudo apt install ffmpeg` (Ensure it's a recent version usually 5.0+)
2. **ab-av1**: The magic tool that calculates VMAF scores.
- Requires Rust.
- Linux/Mac: `curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh` then `cargo install ab-av1`
- Windows: Download binary from https://github.com/alexheretic/ab-av1/releases or use cargo.
## Usage
1. Open a terminal.
2. Run the script pointing to your library folder:
```bash
python3 optimize_library.py "C:\Path\To\Movies"
```
or on Linux:
```bash
python3 optimize_library.py /mnt/media/Movies
```
## Options
- `--vmaf 95`: (Default) "Visually Lossless" target. Matches original source indistinguishably.
- `--vmaf 93`: Good for saving more space while keeping high quality.
- `--preset 6`: (Default) Best balance of speed/quality. Use 4 for archival (slower), 8 for speed.
## Hardware Note
This script is designed to run on your **i9-12900H (MS01)**.
- It uses the CPU (SVT-AV1) which is extremely efficient on the 12900H.
- It is faster to encode locally than to upload/download 50GB files on a 30Mbps connection.
## How it works
1. It scans your folders for videos.
2. It takes a small sample of each video.
3. It tries different compression levels (CRF) to find the highest number (smallest size) that hits VMAF 95.
4. It encodes the full file using that perfect setting.
5. Audio and Subtitles are COPIED (preserved) to ensure no loss of surround sound/metadata.

407
SETUP.md Normal file
View File

@@ -0,0 +1,407 @@
# VMAF Optimisation Pipeline - Setup Guide
## Quick Start
### Prerequisites
- Python 3.8+
- FFmpeg with VMAF support (`ffmpeg -filters 2>&1 | grep libvmaf`)
- ab-av1 binary (v0.10.3+)
### Installation
**On Linux (Server/WSL):**
```bash
# Clone repository
git clone https://gitea.theflagroup.com/bnair/VMAFOptimiser.git /opt/Optmiser
# Download ab-av1
cd /opt/Optmiser/bin
wget https://github.com/alexheretic/ab-av1/releases/download/v0.10.3/ab-av1-x86_64-unknown-linux-musl
chmod +x ab-av1
# Set up directories
mkdir -p /opt/Optmiser/tmp /opt/Optmiser/logs /opt/Optmiser/.lock
```
**On Windows:**
```powershell
# Clone repository
git clone https://gitea.theflagroup.com/bnair/VMAFOptimiser.git C:\Optmiser
# Install dependencies
winget install ffmpeg
# Install Rust and ab-av1
# Download from https://rust-lang.org/
cargo install ab-av1
# Set up directories
mkdir C:\Optmiser\tmp, C:\Optmiser\logs
```
**On macOS:**
```bash
# Clone repository
git clone https://gitea.theflagroup.com/bnair/VMAFOptimiser.git /opt/Optmiser
# Install dependencies
brew install ffmpeg
# Install Rust and ab-av1
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
cargo install ab-av1
# Set up directories
mkdir -p /opt/Optmiser/tmp /opt/Optmiser/logs /opt/Optmiser/.lock
```
## Running the Optimiser
### Choose Your Script
**Linux / WSL / macOS:**
```bash
# Interactive mode (shows prompts)
./run_optimisation.sh
# Direct execution with parameters
./run_optimisation.sh --directory /mnt/Media/Movies --vmaf 95 --preset 6 --workers 1
# For 50% CPU mode on server
./run_optimisation.sh --directory /mnt/Media/Movies --vmaf 95 --workers 1 --cpu-limit 50
```
**Windows:**
```powershell
# Interactive mode (shows prompts)
.\run_optimisation.ps1
# Direct execution with parameters
.\run_optimisation.ps1 -directory "D:\Movies" --vmaf 95 --preset 6 --workers 1
# For hardware acceleration (AMD GPU)
.\run_optimisation.ps1 -directory "D:\Movies" --vmaf 95 --hwaccel auto
```
## Script Parameters
All wrapper scripts (`run_optimisation.sh` on Linux, `run_optimisation.ps1` on Windows) accept these parameters:
| Parameter | Description | Default |
|------------|-------------|---------|
| `--directory <path>` | Root directory to scan | Current directory |
| `--vmaf <score>` | Target VMAF score | 95.0 |
| `--preset <value>` | SVT-AV1 Preset (4=best, 6=balanced, 8=fast) | 6 |
| `--workers <count>` | Concurrent files to process | 1 |
| `--samples <count>` | Samples for CRF search | 4 |
| `--thorough` | Use thorough mode (slower, more accurate) | false |
| `--encoder <name>` | ab-av1 encoder | svt-av1 |
| `--hwaccel <value>` | Hardware acceleration | none (auto: auto-detect) |
## Multi-Machine Setup
### How Lock Files Prevent Conflicts
Each video file has a lock file: `/opt/Optmiser/.lock/{video_filename}`
**Process:**
1. Machine A: Checks for lock → Not found, creates lock
2. Machine A: Starts encoding
3. Machine B: Checks for lock → Found, skips file
4. Machine A: Finishes, removes lock
5. Machine B: Can now process that file
**Result:** Different machines automatically process different files simultaneously!
### Setup for Multiple Machines
**Machine 1 - Linux Server (Intel i9-12900H):**
```bash
cd /opt/Optmiser
git pull origin main
./run_optimisation.sh /mnt/Media/movies --vmaf 95
```
**Machine 2 - Windows PC (AMD RX 7900 XT):**
```powershell
cd C:\Optmiser
git pull origin main
.\run_optimisation.ps1 D:\Media\movies --vmaf 95 --hwaccel auto
```
**Machine 3 - Another Linux PC:**
```bash
cd /opt/Optmiser
git pull origin main
./run_optimisation.sh /home/user/Media/tv --vmaf 95
```
All three can run simultaneously!
## Hardware Acceleration
### Automatic Detection
When `--hwaccel auto` is specified, the wrapper scripts automatically select the best available hardware acceleration:
| Platform | Auto Selection | Notes |
|-----------|----------------|--------|
| Windows | d3d11va | Direct3D Video Acceleration |
| macOS | videotoolbox | VideoToolbox framework |
| Linux/WSL | vaapi | Video Acceleration via VA-API |
### GPU vs iGPU Priority
- **Discrete GPU takes priority:** If a discrete GPU (like AMD RX 7900 XT) is present, it's selected over integrated GPU
- **For AMD RX 7900 XT:** Hardware encoding provides ~3-10x speedup over CPU
- **Note:** GPU encoding may need slightly lower VMAF targets to match CPU quality
## Customization
### Changing VMAF Targets
Edit `optimize_library.py`:
```python
# More aggressive (smaller files, lower quality)
TARGETS = [92.0, 90.0, 88.0]
# Conservative (larger files, higher quality)
TARGETS = [95.0, 94.0, 93.0]
```
### Changing Savings Threshold
```python
# More aggressive (encode more)
MIN_SAVINGS_PERCENT = 8.0
# Less aggressive (encode fewer)
MIN_SAVINGS_PERCENT = 15.0
```
### Changing Encoder Preset
```python
# Faster encodes (larger files, lower quality)
PRESET = 8
# Better quality (slower encodes, smaller files)
PRESET = 4
```
## Platform-Specific Tips
### For Linux Servers (Intel i9-12900H)
1. **Use 50% CPU mode** if running other services:
```bash
./run_optimisation.sh --directory /media --vmaf 95 --workers 1 --cpu-limit 50
```
2. **Run during off-peak hours** to minimize user impact
3. **Monitor CPU temperature:**
```bash
watch -n 2 'sensors | grep "Package id"'
```
4. **Use higher preset for faster encodes:**
```bash
./run_optimisation.sh --vmaf 93 --preset 8
```
### For Windows PCs (AMD RX 7900 XT)
1. **Enable hardware acceleration** for massive speedup:
```powershell
.\run_optimisation.ps1 --directory "D:\Movies" --hwaccel auto
```
2. **Test small sample first** to verify settings:
```powershell
.\run_optimisation.ps1 --directory "D:\Media\sample" --thorough --vmaf 95
```
3. **Monitor GPU usage** (Task Manager or third-party tools)
4. **Consider quality compensation** - GPU encoding may need slightly lower VMAF targets to match CPU quality
### For WSL (Ubuntu/Debian)
1. **Access Windows drives** via `/mnt/c/`:
```bash
ls /mnt/c/Media/movies
```
2. **Increase memory limits** if encoding 4K content:
```bash
# Edit ~/.wslconfig
[wsl2]
memory=16GB
```
## Running in Docker (Optional)
```bash
# Build image
docker build -t vmaf-optimiser .
# Run with mount
docker run -v /path/to/media:/media vmaf-optimiser /media
```
## Troubleshooting
### Issue: Scripts not found
**Solution:** Ensure you're in the correct directory with the scripts installed.
### Issue: "ab-av1: command not found"
**Solution:** Install ab-av1 via cargo:
```bash
cargo install ab-av1
```
### Issue: FFmpeg VMAF not available
**Solution:** Recompile FFmpeg with VMAF support or download a pre-built version that includes libvmaf.
### Issue: Out of Memory
**Solution:** Reduce workers or increase swap:
```bash
# Increase swap
sudo fallocate -l 4G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
# Use more conservative settings
./run_optimisation.sh --workers 1 --vmaf 93
```
### Issue: Multiple machines encoding same file
**Solution:** This is prevented by lock files. If you see duplicates, check `/opt/Optmiser/.lock/` for stale locks.
### Issue: Encoding fails
**Solution:** Check logs:
```bash
cat /opt/Optmiser/logs/failed_encodes.jsonl | jq '.'
```
### Issue: "unexpected argument" error
**Solution:** Use correct flags for your ab-av1 version. The wrapper scripts now validate support at runtime.
## Git Workflow
### Initial Setup
```bash
cd /opt/Optmiser
git init
git remote add origin https://gitea.theflagroup.com/bnair/VMAFOptimiser.git
git branch -M main
git add .
git commit -m "Initial commit: VMAF optimisation pipeline"
git push -u origin main
```
### Daily Updates
```bash
cd /opt/Optmiser
git pull origin main
# Run optimisation
./run_optimisation.sh /media tv_movies
# Review changes
git diff
```
### Committing Changes
```bash
cd /opt/Optmiser
git status
# Add changed files
git add optimize_library.py run_optimisation.sh run_optimisation.ps1
# Commit with message
git commit -m "feat: add Windows and Linux wrapper scripts"
# Push
git push
```
### Viewing Logs
```bash
# Watch logs in real-time
tail -f /opt/Optmiser/logs/tv_movies.jsonl | jq '.'
# Check files logged for review
cat /opt/Optmiser/logs/low_savings_skips.jsonl | jq '.[] | select(.recommendations=="logged_for_review")'
# Statistics
jq -r '.status' /opt/Optmiser/logs/tv_movies.jsonl | sort | uniq -c
# Find what CRF/VMAF combinations are being used most
jq -r '[.vmaf, .crf] | @tsv' /opt/Optmiser/logs/tv_movies.jsonl | sort | uniq -c
```
### View History
```bash
cd /opt/Optmiser
git log --oneline
git log --graph --all
```
## FAQ
**Q: Can I run this on multiple machines at once?**
A: Yes! Each machine will process different files due to lock file mechanism.
**Q: Should I use Windows or WSL?**
A: WSL is recommended for Linux compatibility. Use Windows native if you need direct hardware access or performance.
**Q: Will hardware encoding work better than CPU?**
A: For AMD RX 7900 XT, hardware AV1 encoding is ~3-10x faster than CPU. However, GPU encoding may need slightly lower VMAF targets to match CPU quality.
**Q: What VMAF target should I use?**
A: Start with VMAF 94 or 95. Drop to 92-90 if you need more savings.
**Q: How do I know which files are being processed?**
A: Check `.lock/` directory: `ls -la /opt/Optmiser/.lock/`
**Q: Can I pause/resume?**
A: Pause by stopping the script (Ctrl+C). Resume by running again - it skips processed files.
**Q: What happens if encoding fails?**
A: Error is logged to `failed_encodes.jsonl`. Original file is NOT modified.
**Q: How much CPU does encoding use?**
A: Full CPU by default. Use `--workers 1` for single-threaded, or limit with `--cpu-limit 50` for 50% (12 threads on 24-core).
**Q: What are the log files?**
A:
- `tv_movies.jsonl` - Successful TV & Movie encodes
- `content.jsonl` - Successful Content folder encodes
- `low_savings_skips.jsonl` - Files with <12% savings + 15% estimates
- `failed_searches.jsonl` - Files that couldn't hit any VMAF target
- `failed_encodes.jsonl` - Encoding errors
**Q: How do I see real-time progress?**
A: The script streams all ab-av1 output in real-time, showing ETA, encoding speed, and frame count.
---
**Last Updated:** December 31, 2025
**Version:** 2.0 with Windows and Linux Wrapper Scripts

341
optimize_library.py Normal file
View File

@@ -0,0 +1,341 @@
import os
import sys
import subprocess
import argparse
import json
import shutil
import platform
from concurrent.futures import ThreadPoolExecutor, as_completed
from pathlib import Path
DEFAULT_VMAF = 95.0
DEFAULT_PRESET = 6
DEFAULT_WORKERS = 1
DEFAULT_SAMPLES = 4
EXTENSIONS = {".mkv", ".mp4", ".mov", ".avi", ".ts"}
_AB_AV1_HELP_CACHE = {}
def check_dependencies():
missing = []
for tool in ["ffmpeg", "ffprobe", "ab-av1"]:
if not shutil.which(tool):
missing.append(tool)
if missing:
print(f"Error: Missing required tools: {', '.join(missing)}")
print(
"Please install FFmpeg and 'ab-av1' (via cargo install ab-av1) before running."
)
sys.exit(1)
def is_wsl():
if os.environ.get("WSL_DISTRO_NAME"):
return True
try:
with open("/proc/sys/kernel/osrelease", "r", encoding="utf-8") as f:
return "microsoft" in f.read().lower()
except FileNotFoundError:
return False
def platform_label():
system = platform.system()
if system == "Linux" and is_wsl():
return "Linux (WSL)"
return system
def _ab_av1_help(subcommand):
cached = _AB_AV1_HELP_CACHE.get(subcommand)
if cached is not None:
return cached
try:
result = subprocess.run(
["ab-av1", subcommand, "--help"],
capture_output=True,
text=True,
check=False,
)
help_text = (result.stdout or "") + "\n" + (result.stderr or "")
except Exception:
help_text = ""
_AB_AV1_HELP_CACHE[subcommand] = help_text
return help_text
def ab_av1_supports(subcommand, flag):
return flag in _ab_av1_help(subcommand)
def normalize_hwaccel(value):
if value is None:
return None
v = value.strip()
if not v:
return None
v_lower = v.lower()
if v_lower in {"none", "off", "false", "0"}:
return None
if v_lower != "auto":
return v
system = platform.system()
if system == "Windows":
return "d3d11va"
if system == "Darwin":
return "videotoolbox"
return "vaapi"
def get_video_info(filepath):
try:
cmd = [
"ffprobe",
"-v",
"quiet",
"-print_format",
"json",
"-show_streams",
"-select_streams",
"v:0",
filepath,
]
result = subprocess.run(cmd, capture_output=True, text=True, check=True)
data = json.loads(result.stdout)
streams = data.get("streams") or []
if not streams:
return None
stream = streams[0]
codec = stream.get("codec_name", "unknown")
color_transfer = stream.get("color_transfer", "unknown")
is_hdr = color_transfer in ["smpte2084", "arib-std-b67"]
return {"codec": codec, "is_hdr": is_hdr}
except Exception as e:
print(f"Error probing {filepath}: {e}")
return None
def build_ab_av1_command(input_path, output_path, args):
cmd = [
"ab-av1",
"auto-encode",
"-i",
str(input_path),
"-o",
str(output_path),
"--min-vmaf",
str(args.vmaf),
"--preset",
str(args.preset),
]
if args.encoder:
if ab_av1_supports("auto-encode", "--encoder"):
cmd.extend(["--encoder", args.encoder])
elif ab_av1_supports("auto-encode", "-e"):
cmd.extend(["-e", args.encoder])
else:
print("Warning: This ab-av1 version does not support --encoder; ignoring.")
if args.samples is not None:
if ab_av1_supports("auto-encode", "--samples"):
cmd.extend(["--samples", str(args.samples)])
elif ab_av1_supports("auto-encode", "--sample-count"):
cmd.extend(["--sample-count", str(args.samples)])
else:
print("Warning: This ab-av1 version does not support --samples; ignoring.")
if args.thorough:
if ab_av1_supports("auto-encode", "--thorough"):
cmd.append("--thorough")
else:
print("Warning: This ab-av1 version does not support --thorough; ignoring.")
hwaccel = normalize_hwaccel(args.hwaccel)
if hwaccel is not None:
if ab_av1_supports("auto-encode", "--enc-input"):
cmd.extend(["--enc-input", f"hwaccel={hwaccel}"])
hwaccel_output_format = args.hwaccel_output_format
if hwaccel_output_format is None and hwaccel == "vaapi":
hwaccel_output_format = "vaapi"
if hwaccel_output_format is not None:
cmd.extend(
["--enc-input", f"hwaccel_output_format={hwaccel_output_format}"]
)
else:
print(
"Warning: This ab-av1 version does not support --enc-input; ignoring --hwaccel."
)
if ab_av1_supports("auto-encode", "--acodec"):
cmd.extend(["--acodec", "copy"])
elif ab_av1_supports("auto-encode", "--ac"):
cmd.extend(["--ac", "copy"])
else:
print(
"Warning: This ab-av1 version does not support --acodec/--ac; leaving audio defaults."
)
return cmd
def process_file(filepath, args):
input_path = Path(filepath)
output_path = input_path.with_stem(input_path.stem + "_av1")
if output_path.exists():
print(f"Skipping (Output exists): {input_path.name}")
return
info = get_video_info(str(input_path))
if not info:
return
if info["codec"] == "av1":
print(f"Skipping (Already AV1): {input_path.name}")
return
print(f"\nProcessing: {input_path.name}")
print(f" Source Codec: {info['codec']}")
print(f" HDR: {info['is_hdr']}")
cmd = build_ab_av1_command(input_path, output_path, args)
try:
subprocess.run(cmd, check=True)
print(f"Success! Encoded: {output_path.name}")
except subprocess.CalledProcessError:
print(f"Failed to encode: {input_path.name}")
if output_path.exists():
os.remove(output_path)
def scan_library(root):
files = []
for dirpath, _, filenames in os.walk(root):
for filename in filenames:
if Path(filename).suffix.lower() not in EXTENSIONS:
continue
full_path = Path(dirpath) / filename
if "_av1" in full_path.stem:
continue
files.append(full_path)
return files
def main():
parser = argparse.ArgumentParser(
description="Optimize video library to AV1 using VMAF targeting."
)
parser.add_argument("directory", help="Root directory to scan")
parser.add_argument(
"--vmaf",
type=float,
default=DEFAULT_VMAF,
help=f"Target VMAF score (default: {DEFAULT_VMAF})",
)
parser.add_argument(
"--preset",
type=int,
default=DEFAULT_PRESET,
help=f"SVT-AV1 Preset (default: {DEFAULT_PRESET})",
)
parser.add_argument(
"--workers",
type=int,
default=DEFAULT_WORKERS,
help=f"Concurrent files to process (default: {DEFAULT_WORKERS})",
)
parser.add_argument(
"--samples",
type=int,
default=DEFAULT_SAMPLES,
help=f"Samples to use for CRF search if supported (default: {DEFAULT_SAMPLES})",
)
parser.add_argument(
"--thorough",
action="store_true",
help="Use ab-av1 thorough mode if supported (slower, more accurate)",
)
parser.add_argument(
"--encoder",
default="svt-av1",
help="ab-av1 encoder (default: svt-av1). For AMD AV1 on Windows try: av1_amf",
)
parser.add_argument(
"--hwaccel",
default=None,
help=(
"Hardware acceleration for decode (passed via ab-av1 --enc-input if supported). "
"Examples: auto, vaapi, d3d11va, videotoolbox. Use 'none' to disable."
),
)
parser.add_argument(
"--hwaccel-output-format",
default=None,
help="Optional hwaccel_output_format override (e.g., vaapi)",
)
args = parser.parse_args()
if args.workers < 1:
print("Error: --workers must be >= 1")
sys.exit(2)
check_dependencies()
root = Path(args.directory)
if not root.exists():
print(f"Directory not found: {root}")
sys.exit(1)
print(f"Platform: {platform_label()}")
print(f"Scanning library: {root}")
print(f"Target VMAF: {args.vmaf}")
print(f"Encoder Preset: {args.preset}")
print(f"Workers: {args.workers}")
print(f"Samples: {args.samples}")
print(f"Encoder: {args.encoder}")
if args.hwaccel is not None:
print(f"HWAccel: {args.hwaccel}")
print("-" * 50)
files = scan_library(root)
if not files:
print("No media files found.")
return
if args.workers == 1:
for file_path in files:
process_file(file_path, args)
return
with ThreadPoolExecutor(max_workers=args.workers) as executor:
futures = [
executor.submit(process_file, file_path, args) for file_path in files
]
for future in as_completed(futures):
try:
future.result()
except Exception as e:
print(f"Unexpected error: {e}")
if __name__ == "__main__":
main()

87
run_optimisation.ps1 Normal file
View File

@@ -0,0 +1,87 @@
param(
[Parameter(Mandatory=$false)]
[string]$Directory = ".",
[float]$Vmaf = 95.0,
[int]$Preset = 6,
[int]$Workers = 1,
[int]$Samples = 4,
[switch]$Thorough,
[string]$Encoder = "svt-av1",
[string]$Hwaccel
)
$ErrorActionPreference = "Stop"
function Write-ColorOutput {
param([string]$Message, [string]$Color = "White")
Write-Host $Message -ForegroundColor $Color
}
function Invoke-OptimizeLibrary {
$scriptPath = Join-Path $PSScriptRoot "optimize_library.py"
if (-not (Test-Path $scriptPath)) {
Write-ColorOutput -Message "ERROR: optimize_library.py not found in current directory" -Color "Red"
exit 1
}
$pythonCmd = Get-Command python3, python, py | Select-Object -FirstProperty Path -ErrorAction SilentlyContinue
if (-not $pythonCmd) {
Write-ColorOutput -Message "ERROR: Python 3 not found. Please install Python 3." -Color "Red"
exit 1
}
$arguments = @(
$scriptPath,
$Directory,
"--vmaf", $Vmaf.ToString("F1"),
"--preset", $Preset.ToString(),
"--workers", $Workers.ToString(),
"--samples", $Samples.ToString()
"--encoder", $Encoder
)
if ($Thorough) {
$arguments += "--thorough"
}
if ($Hwaccel) {
$arguments += "--hwaccel", $Hwaccel
}
Write-ColorOutput -Message "Running optimize_library.py..." -Color "Cyan"
Write-ColorOutput -Message " Directory: $Directory" -Color "White"
Write-ColorOutput -Message " Target VMAF: $Vmaf" -Color "White"
Write-ColorOutput -Message " Preset: $Preset" -Color "White"
Write-ColorOutput -Message " Workers: $Workers" -Color "White"
Write-ColorOutput -Message " Samples: $Samples" -Color "White"
Write-ColorOutput -Message " Encoder: $Encoder" -Color "White"
if ($Thorough) {
Write-ColorOutput -Message " Thorough: Yes" -Color "White"
}
if ($Hwaccel) {
Write-ColorOutput -Message " HW Accel: $Hwaccel" -Color "White"
}
Write-Host ""
$process = Start-Process -FilePath $pythonCmd.Path -ArgumentList $arguments -NoNewWindow -PassThru
$process.WaitForExit()
$exitCode = $process.ExitCode
if ($exitCode -eq 0) {
Write-ColorOutput -Message "SUCCESS: Library optimization completed" -Color "Green"
} else {
Write-ColorOutput -Message "ERROR: optimize_library.py exited with code $exitCode" -Color "Red"
}
exit $exitCode
}
Write-ColorOutput -Message "========================================" -Color "Cyan"
Write-ColorOutput -Message "VMAF Library Optimiser (Windows)" -Color "Yellow"
Write-ColorOutput -Message "========================================" -Color "Cyan"
Write-Host ""
Invoke-OptimizeLibrary

151
run_optimisation.sh Normal file
View File

@@ -0,0 +1,151 @@
#!/bin/bash
# VMAF Library Optimiser (Linux/Server runner)
# This script wraps optimize_library.py with the same interface as the Windows PowerShell version
set -e
COLOR_RED='\033[0;31m'
COLOR_GREEN='\033[0;32m'
COLOR_CYAN='\033[0;36m'
COLOR_YELLOW='\033[1;33m'
COLOR_WHITE='\033[0;37m'
COLOR_RESET='\033[0m'
log_info() {
echo -e "${COLOR_CYAN}$*${COLOR_RESET}"
}
log_error() {
echo -e "${COLOR_RED}ERROR: $*${COLOR_RESET}" >&2
}
log_success() {
echo -e "${COLOR_GREEN}$*${COLOR_RESET}"
}
# Default values matching optimize_library.py defaults
DIRECTORY="."
VMAF="95.0"
PRESET="6"
WORKERS="1"
SAMPLES="4"
THOROUGH=""
ENCODER="svt-av1"
HWACCEL=""
# Parse command line arguments
while [[ $# -gt 0 ]]; do
case "$1" in
--directory)
DIRECTORY="$2"
shift 2
;;
--vmaf)
VMAF="$2"
shift 2
;;
--preset)
PRESET="$2"
shift 2
;;
--workers)
WORKERS="$2"
shift 2
;;
--samples)
SAMPLES="$2"
shift 2
;;
--thorough)
THOROUGH="--thorough"
shift
;;
--encoder)
ENCODER="$2"
shift 2
;;
--hwaccel)
HWACCEL="$2"
shift 2
;;
*)
DIRECTORY="$1"
shift
;;
esac
done
# Check if python3 is available
if ! command -v python3 &> /dev/null; then
if ! command -v python &> /dev/null; then
log_error "Python 3 not found. Please install Python 3."
exit 1
else
PYTHON_CMD="python"
fi
else
PYTHON_CMD="python3"
fi
# Check if optimize_library.py exists
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
SCRIPT_PATH="$SCRIPT_DIR/optimize_library.py"
if [[ ! -f "$SCRIPT_PATH" ]]; then
log_error "optimize_library.py not found in: $SCRIPT_DIR"
exit 1
fi
# Build command arguments
ARGS=(
"$PYTHON_CMD" "$SCRIPT_PATH"
"$DIRECTORY"
--vmaf "$VMAF"
--preset "$PRESET"
--workers "$WORKERS"
--samples "$SAMPLES"
--encoder "$ENCODER"
)
if [[ -n "$THOROUGH" ]]; then
ARGS+=(--thorough)
fi
if [[ -n "$HWACCEL" ]]; then
ARGS+=(--hwaccel "$HWACCEL")
fi
# Print configuration
log_info "========================================"
log_info "VMAF Library Optimiser (Linux/Server)"
log_info "========================================"
echo ""
log_info "Directory: $DIRECTORY"
log_info "Target VMAF: $VMAF"
log_info "Preset: $PRESET"
log_info "Workers: $WORKERS"
log_info "Samples: $SAMPLES"
log_info "Encoder: $ENCODER"
if [[ -n "$THOROUGH" ]]; then
log_info "Thorough: Yes"
fi
if [[ -n "$HWACCEL" ]]; then
log_info "HW Accel: $HWACCEL"
fi
echo ""
log_info "Running optimize_library.py..."
echo ""
# Run the optimisation
"${ARGS[@]}"
EXIT_CODE=$?
# Handle exit code
if [ $EXIT_CODE -eq 0 ]; then
log_success "SUCCESS: Library optimisation completed"
else
log_error "optimize_library.py exited with code $EXIT_CODE"
fi
exit $EXIT_CODE