Initial commit: VMAF optimisation pipeline
This commit is contained in:
bnair123
487
SETUP.md
Normal file
487
SETUP.md
Normal file
@@ -0,0 +1,487 @@
|
||||
# VMAF Optimisation Pipeline - Setup Guide
|
||||
|
||||
## Quick Start (Server)
|
||||
|
||||
### Prerequisites
|
||||
- FFmpeg with VMAF support (`ffmpeg -filters 2>&1 | grep libvmaf`)
|
||||
- Python 3.8+
|
||||
- ab-av1 binary (v0.10.3+)
|
||||
|
||||
### Installation
|
||||
```bash
|
||||
# Clone repository
|
||||
git clone https://gitea.theflagroup.com/bnair/VMAFOptimiser.git /opt/Optmiser
|
||||
|
||||
# Download ab-av1 (if not present)
|
||||
cd /opt/Optmiser/bin
|
||||
wget https://github.com/alexheretic/ab-av1/releases/download/v0.10.3/ab-av1-x86_64-unknown-linux-musl
|
||||
chmod +x ab-av1
|
||||
|
||||
# Set up temporary directories
|
||||
mkdir -p /opt/Optmiser/tmp /opt/Optmiser/logs /opt/Optmiser/.lock
|
||||
```
|
||||
|
||||
### Basic Usage
|
||||
```bash
|
||||
# Process TV shows
|
||||
python3 /opt/Optmiser/optimise_media_v2.py /mnt/Media/tv tv_movies
|
||||
|
||||
# Process movies
|
||||
python3 /opt/Optmiser/optimise_media_v2.py /mnt/Media/movies tv_movies
|
||||
|
||||
# Process with 50% CPU limit (leaves CPU for other tasks)
|
||||
python3 /opt/Optmiser/optimise_media_v2.py /mnt/Media/tv tv_movies --cpu-limit 50
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Quick Start (Windows/WSL)
|
||||
|
||||
### WSL Installation (Recommended)
|
||||
```bash
|
||||
# Update WSL
|
||||
wsl --update
|
||||
|
||||
# Install dependencies in WSL Ubuntu/Debian
|
||||
sudo apt update
|
||||
sudo apt install -y ffmpeg python3 git
|
||||
|
||||
# Clone repository into WSL
|
||||
git clone https://gitea.theflagroup.com/bnair/VMAFOptimiser.git /opt/Optmiser
|
||||
|
||||
# Install ab-av1
|
||||
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
|
||||
source ~/.cargo/env
|
||||
cargo install ab-av1
|
||||
|
||||
# Access Windows media from WSL
|
||||
# Windows C:\ is at /mnt/c/
|
||||
python3 /opt/Optmiser/optimise_media_v2.py /mnt/c/Media/tv tv_movies
|
||||
```
|
||||
|
||||
### Native Windows Installation
|
||||
```powershell
|
||||
# Install Python (if not present)
|
||||
# Download from python.org
|
||||
|
||||
# Install FFmpeg
|
||||
winget install ffmpeg
|
||||
|
||||
# Install Rust
|
||||
# Download from rust-lang.org
|
||||
|
||||
# Install ab-av1
|
||||
cargo install ab-av1
|
||||
|
||||
# Clone repository
|
||||
git clone https://gitea.theflagroup.com/bnair/VMAFOptimiser.git C:\Optmiser
|
||||
|
||||
# Run
|
||||
python C:\Optmiser\optimise_media_v2.py D:\Media\tv tv_movies
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Running on Multiple Machines
|
||||
|
||||
### How Lock Files Work
|
||||
|
||||
Each video file has a corresponding lock file:
|
||||
```
|
||||
/opt/Optmiser/.lock/{video_filename}.lock
|
||||
```
|
||||
|
||||
**Process:**
|
||||
1. Machine A checks for lock → None found, creates lock
|
||||
2. Machine A starts encoding
|
||||
3. Machine B checks for lock → Found, skips file
|
||||
4. Machine A finishes, removes lock
|
||||
5. Machine B can now process that file
|
||||
|
||||
**Result:** Different machines automatically process different files!
|
||||
|
||||
### Setup for Multiple Machines
|
||||
|
||||
**Machine 1 - Remote Server (Intel i9-12900H):**
|
||||
```bash
|
||||
cd /opt/Optmiser
|
||||
git pull origin main
|
||||
|
||||
# Run on /mnt/Media (Linux filesystem)
|
||||
python3 optimise_media_v2.py /mnt/Media/tv tv_movies
|
||||
python3 optimise_media_v2.py /mnt/Media/movies tv_movies
|
||||
|
||||
# With 50% CPU limit (recommended)
|
||||
python3 optimise_media_v2.py /mnt/Media/tv tv_movies --cpu-limit 50
|
||||
```
|
||||
|
||||
**Machine 2 - Local PC (AMD RX 7900 XT, Windows):**
|
||||
|
||||
Option A - Native Windows:
|
||||
```powershell
|
||||
# Map network drive if needed
|
||||
# \\Server\media\ or use local storage
|
||||
|
||||
cd C:\Optmiser
|
||||
git pull origin main
|
||||
|
||||
# Run on local media
|
||||
python optimise_media_v2.py D:\Media\tv tv_movies
|
||||
```
|
||||
|
||||
Option B - WSL (Recommended):
|
||||
```bash
|
||||
# Windows C: drive accessible at /mnt/c/
|
||||
cd /opt/Optmiser
|
||||
python optimise_media_v2.py /mnt/c/Media/tv tv_movies
|
||||
```
|
||||
|
||||
**Machine 3 - Another PC (AMD 9800X3D, Linux):**
|
||||
```bash
|
||||
cd /opt/Optmiser
|
||||
git pull origin main
|
||||
|
||||
# Run on local media directory
|
||||
python optimise_media_v2.py /home/user/Media/tv tv_movies
|
||||
```
|
||||
|
||||
All three can run simultaneously - lock files prevent duplicates!
|
||||
|
||||
---
|
||||
|
||||
## Hardware Encoding Guide
|
||||
|
||||
### Detecting Hardware
|
||||
|
||||
The script automatically checks for:
|
||||
1. **AMD GPU encoding support** (future feature)
|
||||
2. **System thread count**
|
||||
3. **Operating system** (Linux vs Windows)
|
||||
|
||||
### CPU Encoding (Software)
|
||||
|
||||
**Best for:** Servers, multi-tasking
|
||||
|
||||
```bash
|
||||
# Force CPU encoding
|
||||
python3 optimise_media_v2.py /media --use-cpu
|
||||
|
||||
# Limit to 50% of CPU (12 threads on 24-core)
|
||||
python3 optimise_media_v2.py /media --cpu-limit 50
|
||||
|
||||
# Set specific worker count
|
||||
python3 optimise_media_v2.py /media --workers 8
|
||||
```
|
||||
|
||||
**When to use:**
|
||||
- Running Plex/Jellyfin on same machine
|
||||
- Server environment with multiple users
|
||||
- Want to leave GPU free for other tasks
|
||||
|
||||
**Expected Speed:** 8-15 fps @ 1080p
|
||||
|
||||
### GPU Encoding (AMD RX 7900 XT)
|
||||
|
||||
**Status:** Planned for future versions
|
||||
**Expected Speedup:** 3-10x vs CPU encoding
|
||||
|
||||
```bash
|
||||
# Enable hardware encoding (when implemented)
|
||||
python3 optimise_media_v2.py /media --use-hardware
|
||||
```
|
||||
|
||||
**When to use:**
|
||||
- Local PC with AMD GPU
|
||||
- Want faster encoding speeds
|
||||
- Can compensate quality with slightly lower VMAF target
|
||||
|
||||
**Expected Speed:** 150+ fps @ 1080p
|
||||
|
||||
### Quality Compensation for GPU Encoding
|
||||
|
||||
GPU encoding may need quality adjustments:
|
||||
|
||||
| CPU VMAF | Equivalent GPU VMAF | Why |
|
||||
|------------|----------------------|-----|
|
||||
| 94 | 92-93 | GPU has quality limitations |
|
||||
| 93 | 91-92 | Hardware encoding trade-offs |
|
||||
| 90 | 88-89 | Significant compression |
|
||||
|
||||
**Recommendation:** If using GPU encoding, set `TARGETS = [92.0, 90.0, 88.0]` for similar quality.
|
||||
|
||||
---
|
||||
|
||||
## Windows/WSL Path Mapping
|
||||
|
||||
### Understanding /mnt/c/
|
||||
|
||||
In WSL, Windows drives are mapped:
|
||||
|
||||
| Windows | WSL Path | Example |
|
||||
|---------|------------|---------|
|
||||
| C:\ | /mnt/c/ | /mnt/c/Users/bnair/Downloads |
|
||||
| D:\ | /mnt/d/ | /mnt/d/Movies/ |
|
||||
| E:\ | /mnt/e/ | /mnt/e/TV\ |
|
||||
|
||||
**To access Windows media from WSL:**
|
||||
```bash
|
||||
# List C:\Media\tv
|
||||
ls /mnt/c/Media/tv
|
||||
|
||||
# Process from WSL
|
||||
python3 /opt/Optmiser/optimise_media_v2.py /mnt/c/Media/tv tv_movies
|
||||
```
|
||||
|
||||
### Network Drives on WSL
|
||||
|
||||
```bash
|
||||
# Map network drive (one-time)
|
||||
sudo mkdir -p /mnt/server
|
||||
echo "//192.168.1.100/media | /mnt/server cifs credentials=/path/to/credfile,uid=1000,gid=1000 0 0" | sudo tee -a /etc/fstab
|
||||
|
||||
# Access network media
|
||||
python3 /opt/Optmiser/optimise_media_v2.py /mnt/server/Media/tv tv_movies
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Customization
|
||||
|
||||
### Changing VMAF Targets
|
||||
|
||||
Edit `optimise_media_v2.py`:
|
||||
|
||||
```python
|
||||
# Line 15
|
||||
TARGETS = [94.0, 93.0, 92.0, 90.0]
|
||||
|
||||
# More aggressive (smaller files, lower quality)
|
||||
TARGETS = [92.0, 90.0, 88.0]
|
||||
|
||||
# Conservative (larger files, higher quality)
|
||||
TARGETS = [95.0, 94.0, 93.0]
|
||||
```
|
||||
|
||||
### Changing Savings Threshold
|
||||
|
||||
```python
|
||||
# Line 17
|
||||
MIN_SAVINGS_PERCENT = 12.0 # Current threshold
|
||||
MIN_SAVINGS_PERCENT = 8.0 # More aggressive (encode more)
|
||||
MIN_SAVINGS_PERCENT = 15.0 # Less aggressive (encode fewer)
|
||||
```
|
||||
|
||||
### Changing Encoder Preset
|
||||
|
||||
```python
|
||||
# Line 19
|
||||
PRESET = 6
|
||||
|
||||
# Faster encodes (larger files, lower quality)
|
||||
PRESET = 8
|
||||
|
||||
# Better quality (slower encodes, smaller files)
|
||||
PRESET = 4
|
||||
```
|
||||
|
||||
### Changing Estimate Target
|
||||
|
||||
```python
|
||||
# Line 18
|
||||
TARGET_SAVINGS_FOR_ESTIMATE = 15.0
|
||||
|
||||
# Target higher savings
|
||||
TARGET_SAVINGS_FOR_ESTIMATE = 20.0
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Monitoring
|
||||
|
||||
### Watch Progress in Real-Time
|
||||
|
||||
```bash
|
||||
# Tail log file with JSON formatting
|
||||
tail -f /opt/Optmiser/logs/tv_movies.jsonl | jq '.'
|
||||
|
||||
# Monitor encoding speed
|
||||
watch -n 5 'jq -r "[.savings, .duration]" /opt/Optmiser/logs/tv_movies.jsonl | tail -1'
|
||||
|
||||
# Check lock files (what's being processed)
|
||||
ls -la /opt/Optmiser/.lock/
|
||||
```
|
||||
|
||||
### Performance Dashboard
|
||||
|
||||
```bash
|
||||
# Create a simple dashboard
|
||||
watch -n 10 '
|
||||
echo "=== VMAF Optimiser Status ==="
|
||||
echo ""
|
||||
echo "Recent Encodes:"
|
||||
tail -3 /opt/Optmiser/logs/tv_movies.jsonl | jq -r "[.file, .savings, .duration] | @tsv"
|
||||
echo ""
|
||||
echo "CPU Usage:"
|
||||
top -bn1 | head -5
|
||||
'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Issue: Scripts not found
|
||||
|
||||
```bash
|
||||
# Check path
|
||||
ls /opt/Optmiser/optimise_media_v2.py
|
||||
|
||||
# Check Python version
|
||||
python3 --version # Should be 3.8+
|
||||
|
||||
# Check ab-av1
|
||||
/opt/Optmiser/bin/ab-av1 --version # Should be 0.10.3+
|
||||
```
|
||||
|
||||
### Issue: Permission denied
|
||||
|
||||
```bash
|
||||
# Make scripts executable
|
||||
chmod +x /opt/Optmiser/*.py
|
||||
chmod +x /opt/Optmiser/*.sh
|
||||
|
||||
# Fix lock directory permissions
|
||||
chmod 777 /opt/Optmiser/.lock
|
||||
```
|
||||
|
||||
### Issue: ab-av1 not found
|
||||
|
||||
```bash
|
||||
# Check if in PATH
|
||||
which ab-av1
|
||||
|
||||
# Use full path if not in PATH
|
||||
/opt/Optmiser/bin/ab-av1 --version
|
||||
|
||||
# Add to PATH
|
||||
export PATH="$PATH:/opt/Optmiser/bin"
|
||||
```
|
||||
|
||||
### Issue: FFmpeg VMAF not available
|
||||
|
||||
```bash
|
||||
# Check libvmaf
|
||||
ffmpeg -filters 2>&1 | grep libvmaf
|
||||
|
||||
# If not found, recompile FFmpeg with VMAF
|
||||
# Or download compiled version from johnvansickle.com/ffmpeg
|
||||
```
|
||||
|
||||
### Issue: Out of Memory
|
||||
|
||||
```bash
|
||||
# Check available memory
|
||||
free -h
|
||||
|
||||
# Reduce workers
|
||||
python3 optimise_media_v2.py /media --workers 4
|
||||
|
||||
# Increase swap (if needed)
|
||||
sudo fallocate -l 4G /swapfile
|
||||
sudo chmod 600 /swapfile
|
||||
sudo mkswap /swapfile
|
||||
sudo swapon /swapfile
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Git Workflow
|
||||
|
||||
### Initial Setup
|
||||
```bash
|
||||
cd /opt/Optmiser
|
||||
git init
|
||||
git remote add origin https://gitea.theflagroup.com/bnair/VMAFOptimiser.git
|
||||
git branch -M main
|
||||
git add .
|
||||
git commit -m "Initial commit: VMAF optimisation pipeline"
|
||||
git push -u origin main
|
||||
```
|
||||
|
||||
### Daily Updates
|
||||
```bash
|
||||
cd /opt/Optmiser
|
||||
git pull origin main
|
||||
|
||||
# Run optimisation
|
||||
python3 optimise_media_v2.py /media tv_movies
|
||||
|
||||
# Review changes
|
||||
git diff
|
||||
```
|
||||
|
||||
### Commit Changes
|
||||
```bash
|
||||
cd /opt/Optmiser
|
||||
git status
|
||||
|
||||
# Add changed files
|
||||
git add optimise_media_v2.py run_optimisation.sh
|
||||
|
||||
# Commit with message
|
||||
git commit -m "feat: add X"
|
||||
|
||||
# Push
|
||||
git push
|
||||
```
|
||||
|
||||
### View History
|
||||
```bash
|
||||
cd /opt/Optmiser
|
||||
git log --oneline
|
||||
git log --graph --all
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## FAQ
|
||||
|
||||
**Q: Can I run this on multiple machines at once?**
|
||||
A: Yes! Each machine will process different files due to lock file mechanism.
|
||||
|
||||
**Q: Will hardware encoding be available?**
|
||||
A: Planned for future versions. Currently uses CPU encoding (software AV1).
|
||||
|
||||
**Q: How do I know if a file is being encoded elsewhere?**
|
||||
A: Check `/opt/Optmiser/.lock/` - if a lock exists, another machine is processing it.
|
||||
|
||||
**Q: Can I change the VMAF target?**
|
||||
A: Yes, edit `TARGETS = [94.0, 93.0, 92.0, 90.0]` in `optimise_media_v2.py`.
|
||||
|
||||
**Q: What if savings are always below 12%?**
|
||||
A: The script will log 15% estimates to `low_savings_skips.jsonl`. Review these logs to decide if encoding is worth it.
|
||||
|
||||
**Q: Does this work on Windows/WSL?**
|
||||
A: Yes! See the Windows/WSL section for setup instructions.
|
||||
|
||||
**Q: How much CPU does encoding use?**
|
||||
A: Full CPU (24 threads) by default. Use `--cpu-limit 50` for 50% mode.
|
||||
|
||||
**Q: Can I pause/resume?**
|
||||
A: Pause by stopping the script (Ctrl+C). Resume by running again - it skips processed files.
|
||||
|
||||
**Q: What happens if encoding fails?**
|
||||
A: Error is logged to `failed_encodes.jsonl` with error code. Original file is NOT modified.
|
||||
|
||||
---
|
||||
|
||||
## Support
|
||||
|
||||
For issues or questions:
|
||||
- Check AGENTS.md for detailed technical documentation
|
||||
- Review logs in `/opt/Optmiser/logs/`
|
||||
- Test with `--dry-run` flag first
|
||||
|
||||
---
|
||||
|
||||
**Last Updated:** December 31, 2025
|
||||
Reference in New Issue
Block a user