feat: add Windows and Linux wrapper scripts for VMAF optimizer
This commit is contained in:
bnair123
407
SETUP.md
Normal file
407
SETUP.md
Normal file
@@ -0,0 +1,407 @@
|
||||
# VMAF Optimisation Pipeline - Setup Guide
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Prerequisites
|
||||
- Python 3.8+
|
||||
- FFmpeg with VMAF support (`ffmpeg -filters 2>&1 | grep libvmaf`)
|
||||
- ab-av1 binary (v0.10.3+)
|
||||
|
||||
### Installation
|
||||
|
||||
**On Linux (Server/WSL):**
|
||||
```bash
|
||||
# Clone repository
|
||||
git clone https://gitea.theflagroup.com/bnair/VMAFOptimiser.git /opt/Optmiser
|
||||
|
||||
# Download ab-av1
|
||||
cd /opt/Optmiser/bin
|
||||
wget https://github.com/alexheretic/ab-av1/releases/download/v0.10.3/ab-av1-x86_64-unknown-linux-musl
|
||||
chmod +x ab-av1
|
||||
|
||||
# Set up directories
|
||||
mkdir -p /opt/Optmiser/tmp /opt/Optmiser/logs /opt/Optmiser/.lock
|
||||
```
|
||||
|
||||
**On Windows:**
|
||||
```powershell
|
||||
# Clone repository
|
||||
git clone https://gitea.theflagroup.com/bnair/VMAFOptimiser.git C:\Optmiser
|
||||
|
||||
# Install dependencies
|
||||
winget install ffmpeg
|
||||
|
||||
# Install Rust and ab-av1
|
||||
# Download from https://rust-lang.org/
|
||||
cargo install ab-av1
|
||||
|
||||
# Set up directories
|
||||
mkdir C:\Optmiser\tmp, C:\Optmiser\logs
|
||||
```
|
||||
|
||||
**On macOS:**
|
||||
```bash
|
||||
# Clone repository
|
||||
git clone https://gitea.theflagroup.com/bnair/VMAFOptimiser.git /opt/Optmiser
|
||||
|
||||
# Install dependencies
|
||||
brew install ffmpeg
|
||||
|
||||
# Install Rust and ab-av1
|
||||
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
|
||||
cargo install ab-av1
|
||||
|
||||
# Set up directories
|
||||
mkdir -p /opt/Optmiser/tmp /opt/Optmiser/logs /opt/Optmiser/.lock
|
||||
```
|
||||
|
||||
## Running the Optimiser
|
||||
|
||||
### Choose Your Script
|
||||
|
||||
**Linux / WSL / macOS:**
|
||||
```bash
|
||||
# Interactive mode (shows prompts)
|
||||
./run_optimisation.sh
|
||||
|
||||
# Direct execution with parameters
|
||||
./run_optimisation.sh --directory /mnt/Media/Movies --vmaf 95 --preset 6 --workers 1
|
||||
|
||||
# For 50% CPU mode on server
|
||||
./run_optimisation.sh --directory /mnt/Media/Movies --vmaf 95 --workers 1 --cpu-limit 50
|
||||
```
|
||||
|
||||
**Windows:**
|
||||
```powershell
|
||||
# Interactive mode (shows prompts)
|
||||
.\run_optimisation.ps1
|
||||
|
||||
# Direct execution with parameters
|
||||
.\run_optimisation.ps1 -directory "D:\Movies" --vmaf 95 --preset 6 --workers 1
|
||||
|
||||
# For hardware acceleration (AMD GPU)
|
||||
.\run_optimisation.ps1 -directory "D:\Movies" --vmaf 95 --hwaccel auto
|
||||
```
|
||||
|
||||
## Script Parameters
|
||||
|
||||
All wrapper scripts (`run_optimisation.sh` on Linux, `run_optimisation.ps1` on Windows) accept these parameters:
|
||||
|
||||
| Parameter | Description | Default |
|
||||
|------------|-------------|---------|
|
||||
| `--directory <path>` | Root directory to scan | Current directory |
|
||||
| `--vmaf <score>` | Target VMAF score | 95.0 |
|
||||
| `--preset <value>` | SVT-AV1 Preset (4=best, 6=balanced, 8=fast) | 6 |
|
||||
| `--workers <count>` | Concurrent files to process | 1 |
|
||||
| `--samples <count>` | Samples for CRF search | 4 |
|
||||
| `--thorough` | Use thorough mode (slower, more accurate) | false |
|
||||
| `--encoder <name>` | ab-av1 encoder | svt-av1 |
|
||||
| `--hwaccel <value>` | Hardware acceleration | none (auto: auto-detect) |
|
||||
|
||||
## Multi-Machine Setup
|
||||
|
||||
### How Lock Files Prevent Conflicts
|
||||
|
||||
Each video file has a lock file: `/opt/Optmiser/.lock/{video_filename}`
|
||||
|
||||
**Process:**
|
||||
1. Machine A: Checks for lock → Not found, creates lock
|
||||
2. Machine A: Starts encoding
|
||||
3. Machine B: Checks for lock → Found, skips file
|
||||
4. Machine A: Finishes, removes lock
|
||||
5. Machine B: Can now process that file
|
||||
|
||||
**Result:** Different machines automatically process different files simultaneously!
|
||||
|
||||
### Setup for Multiple Machines
|
||||
|
||||
**Machine 1 - Linux Server (Intel i9-12900H):**
|
||||
```bash
|
||||
cd /opt/Optmiser
|
||||
git pull origin main
|
||||
./run_optimisation.sh /mnt/Media/movies --vmaf 95
|
||||
```
|
||||
|
||||
**Machine 2 - Windows PC (AMD RX 7900 XT):**
|
||||
```powershell
|
||||
cd C:\Optmiser
|
||||
git pull origin main
|
||||
.\run_optimisation.ps1 D:\Media\movies --vmaf 95 --hwaccel auto
|
||||
```
|
||||
|
||||
**Machine 3 - Another Linux PC:**
|
||||
```bash
|
||||
cd /opt/Optmiser
|
||||
git pull origin main
|
||||
./run_optimisation.sh /home/user/Media/tv --vmaf 95
|
||||
```
|
||||
|
||||
All three can run simultaneously!
|
||||
|
||||
## Hardware Acceleration
|
||||
|
||||
### Automatic Detection
|
||||
|
||||
When `--hwaccel auto` is specified, the wrapper scripts automatically select the best available hardware acceleration:
|
||||
|
||||
| Platform | Auto Selection | Notes |
|
||||
|-----------|----------------|--------|
|
||||
| Windows | d3d11va | Direct3D Video Acceleration |
|
||||
| macOS | videotoolbox | VideoToolbox framework |
|
||||
| Linux/WSL | vaapi | Video Acceleration via VA-API |
|
||||
|
||||
### GPU vs iGPU Priority
|
||||
|
||||
- **Discrete GPU takes priority:** If a discrete GPU (like AMD RX 7900 XT) is present, it's selected over integrated GPU
|
||||
- **For AMD RX 7900 XT:** Hardware encoding provides ~3-10x speedup over CPU
|
||||
- **Note:** GPU encoding may need slightly lower VMAF targets to match CPU quality
|
||||
|
||||
## Customization
|
||||
|
||||
### Changing VMAF Targets
|
||||
|
||||
Edit `optimize_library.py`:
|
||||
|
||||
```python
|
||||
# More aggressive (smaller files, lower quality)
|
||||
TARGETS = [92.0, 90.0, 88.0]
|
||||
|
||||
# Conservative (larger files, higher quality)
|
||||
TARGETS = [95.0, 94.0, 93.0]
|
||||
```
|
||||
|
||||
### Changing Savings Threshold
|
||||
|
||||
```python
|
||||
# More aggressive (encode more)
|
||||
MIN_SAVINGS_PERCENT = 8.0
|
||||
|
||||
# Less aggressive (encode fewer)
|
||||
MIN_SAVINGS_PERCENT = 15.0
|
||||
```
|
||||
|
||||
### Changing Encoder Preset
|
||||
|
||||
```python
|
||||
# Faster encodes (larger files, lower quality)
|
||||
PRESET = 8
|
||||
|
||||
# Better quality (slower encodes, smaller files)
|
||||
PRESET = 4
|
||||
```
|
||||
|
||||
## Platform-Specific Tips
|
||||
|
||||
### For Linux Servers (Intel i9-12900H)
|
||||
|
||||
1. **Use 50% CPU mode** if running other services:
|
||||
```bash
|
||||
./run_optimisation.sh --directory /media --vmaf 95 --workers 1 --cpu-limit 50
|
||||
```
|
||||
|
||||
2. **Run during off-peak hours** to minimize user impact
|
||||
|
||||
3. **Monitor CPU temperature:**
|
||||
```bash
|
||||
watch -n 2 'sensors | grep "Package id"'
|
||||
```
|
||||
|
||||
4. **Use higher preset for faster encodes:**
|
||||
```bash
|
||||
./run_optimisation.sh --vmaf 93 --preset 8
|
||||
```
|
||||
|
||||
### For Windows PCs (AMD RX 7900 XT)
|
||||
|
||||
1. **Enable hardware acceleration** for massive speedup:
|
||||
```powershell
|
||||
.\run_optimisation.ps1 --directory "D:\Movies" --hwaccel auto
|
||||
```
|
||||
|
||||
2. **Test small sample first** to verify settings:
|
||||
```powershell
|
||||
.\run_optimisation.ps1 --directory "D:\Media\sample" --thorough --vmaf 95
|
||||
```
|
||||
|
||||
3. **Monitor GPU usage** (Task Manager or third-party tools)
|
||||
|
||||
4. **Consider quality compensation** - GPU encoding may need slightly lower VMAF targets to match CPU quality
|
||||
|
||||
### For WSL (Ubuntu/Debian)
|
||||
|
||||
1. **Access Windows drives** via `/mnt/c/`:
|
||||
```bash
|
||||
ls /mnt/c/Media/movies
|
||||
```
|
||||
|
||||
2. **Increase memory limits** if encoding 4K content:
|
||||
```bash
|
||||
# Edit ~/.wslconfig
|
||||
[wsl2]
|
||||
memory=16GB
|
||||
```
|
||||
|
||||
## Running in Docker (Optional)
|
||||
|
||||
```bash
|
||||
# Build image
|
||||
docker build -t vmaf-optimiser .
|
||||
|
||||
# Run with mount
|
||||
docker run -v /path/to/media:/media vmaf-optimiser /media
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Issue: Scripts not found
|
||||
|
||||
**Solution:** Ensure you're in the correct directory with the scripts installed.
|
||||
|
||||
### Issue: "ab-av1: command not found"
|
||||
|
||||
**Solution:** Install ab-av1 via cargo:
|
||||
```bash
|
||||
cargo install ab-av1
|
||||
```
|
||||
|
||||
### Issue: FFmpeg VMAF not available
|
||||
|
||||
**Solution:** Recompile FFmpeg with VMAF support or download a pre-built version that includes libvmaf.
|
||||
|
||||
### Issue: Out of Memory
|
||||
|
||||
**Solution:** Reduce workers or increase swap:
|
||||
```bash
|
||||
# Increase swap
|
||||
sudo fallocate -l 4G /swapfile
|
||||
sudo chmod 600 /swapfile
|
||||
sudo mkswap /swapfile
|
||||
sudo swapon /swapfile
|
||||
|
||||
# Use more conservative settings
|
||||
./run_optimisation.sh --workers 1 --vmaf 93
|
||||
```
|
||||
|
||||
### Issue: Multiple machines encoding same file
|
||||
|
||||
**Solution:** This is prevented by lock files. If you see duplicates, check `/opt/Optmiser/.lock/` for stale locks.
|
||||
|
||||
### Issue: Encoding fails
|
||||
|
||||
**Solution:** Check logs:
|
||||
```bash
|
||||
cat /opt/Optmiser/logs/failed_encodes.jsonl | jq '.'
|
||||
```
|
||||
|
||||
### Issue: "unexpected argument" error
|
||||
|
||||
**Solution:** Use correct flags for your ab-av1 version. The wrapper scripts now validate support at runtime.
|
||||
|
||||
## Git Workflow
|
||||
|
||||
### Initial Setup
|
||||
|
||||
```bash
|
||||
cd /opt/Optmiser
|
||||
git init
|
||||
git remote add origin https://gitea.theflagroup.com/bnair/VMAFOptimiser.git
|
||||
git branch -M main
|
||||
git add .
|
||||
git commit -m "Initial commit: VMAF optimisation pipeline"
|
||||
git push -u origin main
|
||||
```
|
||||
|
||||
### Daily Updates
|
||||
|
||||
```bash
|
||||
cd /opt/Optmiser
|
||||
git pull origin main
|
||||
|
||||
# Run optimisation
|
||||
./run_optimisation.sh /media tv_movies
|
||||
|
||||
# Review changes
|
||||
git diff
|
||||
```
|
||||
|
||||
### Committing Changes
|
||||
|
||||
```bash
|
||||
cd /opt/Optmiser
|
||||
git status
|
||||
|
||||
# Add changed files
|
||||
git add optimize_library.py run_optimisation.sh run_optimisation.ps1
|
||||
|
||||
# Commit with message
|
||||
git commit -m "feat: add Windows and Linux wrapper scripts"
|
||||
|
||||
# Push
|
||||
git push
|
||||
```
|
||||
|
||||
### Viewing Logs
|
||||
|
||||
```bash
|
||||
# Watch logs in real-time
|
||||
tail -f /opt/Optmiser/logs/tv_movies.jsonl | jq '.'
|
||||
|
||||
# Check files logged for review
|
||||
cat /opt/Optmiser/logs/low_savings_skips.jsonl | jq '.[] | select(.recommendations=="logged_for_review")'
|
||||
|
||||
# Statistics
|
||||
jq -r '.status' /opt/Optmiser/logs/tv_movies.jsonl | sort | uniq -c
|
||||
|
||||
# Find what CRF/VMAF combinations are being used most
|
||||
jq -r '[.vmaf, .crf] | @tsv' /opt/Optmiser/logs/tv_movies.jsonl | sort | uniq -c
|
||||
```
|
||||
|
||||
### View History
|
||||
|
||||
```bash
|
||||
cd /opt/Optmiser
|
||||
git log --oneline
|
||||
git log --graph --all
|
||||
```
|
||||
|
||||
## FAQ
|
||||
|
||||
**Q: Can I run this on multiple machines at once?**
|
||||
A: Yes! Each machine will process different files due to lock file mechanism.
|
||||
|
||||
**Q: Should I use Windows or WSL?**
|
||||
A: WSL is recommended for Linux compatibility. Use Windows native if you need direct hardware access or performance.
|
||||
|
||||
**Q: Will hardware encoding work better than CPU?**
|
||||
A: For AMD RX 7900 XT, hardware AV1 encoding is ~3-10x faster than CPU. However, GPU encoding may need slightly lower VMAF targets to match CPU quality.
|
||||
|
||||
**Q: What VMAF target should I use?**
|
||||
A: Start with VMAF 94 or 95. Drop to 92-90 if you need more savings.
|
||||
|
||||
**Q: How do I know which files are being processed?**
|
||||
A: Check `.lock/` directory: `ls -la /opt/Optmiser/.lock/`
|
||||
|
||||
**Q: Can I pause/resume?**
|
||||
A: Pause by stopping the script (Ctrl+C). Resume by running again - it skips processed files.
|
||||
|
||||
**Q: What happens if encoding fails?**
|
||||
A: Error is logged to `failed_encodes.jsonl`. Original file is NOT modified.
|
||||
|
||||
**Q: How much CPU does encoding use?**
|
||||
A: Full CPU by default. Use `--workers 1` for single-threaded, or limit with `--cpu-limit 50` for 50% (12 threads on 24-core).
|
||||
|
||||
**Q: What are the log files?**
|
||||
A:
|
||||
- `tv_movies.jsonl` - Successful TV & Movie encodes
|
||||
- `content.jsonl` - Successful Content folder encodes
|
||||
- `low_savings_skips.jsonl` - Files with <12% savings + 15% estimates
|
||||
- `failed_searches.jsonl` - Files that couldn't hit any VMAF target
|
||||
- `failed_encodes.jsonl` - Encoding errors
|
||||
|
||||
**Q: How do I see real-time progress?**
|
||||
A: The script streams all ab-av1 output in real-time, showing ETA, encoding speed, and frame count.
|
||||
|
||||
---
|
||||
|
||||
**Last Updated:** December 31, 2025
|
||||
**Version:** 2.0 with Windows and Linux Wrapper Scripts
|
||||
Reference in New Issue
Block a user