bnair/VMAFOptimiser
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
7c1994c56974552bcf7f7044afb209a952868b4c
VMAFOptimiser/SETUP.md

10 KiB
Raw Blame History

VMAF Optimisation Pipeline - Setup Guide

Quick Start

Prerequisites

  • Python 3.8+
  • FFmpeg with VMAF support (ffmpeg -filters 2>&1 | grep libvmaf)
  • ab-av1 binary (v0.10.3+)

Installation

On Linux (Server/WSL):

91418fa898

# Clone repository
git clone https://gitea.theflagroup.com/bnair/VMAFOptimiser.git /opt/Optmiser

# Download ab-av1
cd /opt/Optmiser/bin
wget https://github.com/alexheretic/ab-av1/releases/download/v0.10.3/ab-av1-x86_64-unknown-linux-musl
chmod +x ab-av1

# Set up directories
mkdir -p /opt/Optmiser/tmp /opt/Optmiser/logs /opt/Optmiser/.lock

On Windows:

# Clone repository
git clone https://gitea.theflagroup.com/bnair/VMAFOptimiser.git C:\Optmiser

# Install dependencies
winget install ffmpeg

# Install Rust and ab-av1
# Download from https://rust-lang.org/
cargo install ab-av1

# Set up directories
mkdir C:\Optmiser\tmp, C:\Optmiser\logs

On macOS:

# Clone repository
git clone https://gitea.theflagroup.com/bnair/VMAFOptimiser.git /opt/Optmiser

# Install dependencies
brew install ffmpeg

# Install Rust and ab-av1
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
cargo install ab-av1

# Set up directories
mkdir -p /opt/Optmiser/tmp /opt/Optmiser/logs /opt/Optmiser/.lock

Running the Optimiser

Choose Your Script

Linux / WSL / macOS:

# Interactive mode (shows prompts)
./run_optimisation.sh

# Direct execution with parameters
./run_optimisation.sh --directory /mnt/Media/Movies --vmaf 95 --preset 6 --workers 1

# For 50% CPU mode on server
./run_optimisation.sh --directory /mnt/Media/Movies --vmaf 95 --workers 1 --cpu-limit 50

Windows:

# Interactive mode (shows prompts)
.\run_optimisation.ps1

# Direct execution with parameters
.\run_optimisation.ps1 -directory "D:\Movies" --vmaf 95 --preset 6 --workers 1

# For hardware acceleration (AMD GPU)
.\run_optimisation.ps1 -directory "D:\Movies" --vmaf 95 --hwaccel auto

Script Parameters

All wrapper scripts (run_optimisation.sh on Linux, run_optimisation.ps1 on Windows) accept these parameters:

Parameter Description Default
--directory <path> Root directory to scan Current directory
--vmaf <score> Target VMAF score 95.0
--preset <value> SVT-AV1 Preset (4=best, 6=balanced, 8=fast) 6
--workers <count> Concurrent files to process 1
--samples <count> Samples for CRF search 4
--thorough Use thorough mode (slower, more accurate) false
--encoder <name> ab-av1 encoder svt-av1
--hwaccel <value> Hardware acceleration none (auto: auto-detect)

Multi-Machine Setup

How Lock Files Prevent Conflicts

Each video file has a lock file: /opt/Optmiser/.lock/{video_filename}

Process:

  1. Machine A: Checks for lock → Not found, creates lock
  2. Machine A: Starts encoding
  3. Machine B: Checks for lock → Found, skips file
  4. Machine A: Finishes, removes lock
  5. Machine B: Can now process that file

Result: Different machines automatically process different files simultaneously!

Setup for Multiple Machines

Machine 1 - Linux Server (Intel i9-12900H):

cd /opt/Optmiser
git pull origin main
./run_optimisation.sh /mnt/Media/movies --vmaf 95

Machine 2 - Windows PC (AMD RX 7900 XT):

cd C:\Optmiser
git pull origin main
.\run_optimisation.ps1 D:\Media\movies --vmaf 95 --hwaccel auto

Machine 3 - Another Linux PC:

cd /opt/Optmiser
git pull origin main
./run_optimisation.sh /home/user/Media/tv --vmaf 95

All three can run simultaneously!

Hardware Acceleration

Automatic Detection

When --hwaccel auto is specified, the wrapper scripts automatically select the best available hardware acceleration:

Platform Auto Selection Notes
Windows d3d11va Direct3D Video Acceleration
macOS videotoolbox VideoToolbox framework
Linux/WSL vaapi Video Acceleration via VA-API

GPU vs iGPU Priority

  • Discrete GPU takes priority: If a discrete GPU (like AMD RX 7900 XT) is present, it's selected over integrated GPU
  • For AMD RX 7900 XT: Hardware encoding provides ~3-10x speedup over CPU
  • Note: GPU encoding may need slightly lower VMAF targets to match CPU quality

Customization

Changing VMAF Targets

Edit optimize_library.py:

# More aggressive (smaller files, lower quality)
TARGETS = [92.0, 90.0, 88.0]

# Conservative (larger files, higher quality)
TARGETS = [95.0, 94.0, 93.0]

Changing Savings Threshold

# More aggressive (encode more)
MIN_SAVINGS_PERCENT = 8.0

# Less aggressive (encode fewer)
MIN_SAVINGS_PERCENT = 15.0

Changing Encoder Preset

# Faster encodes (larger files, lower quality)
PRESET = 8

# Better quality (slower encodes, smaller files)
PRESET = 4

Platform-Specific Tips

For Linux Servers (Intel i9-12900H)

  1. Use 50% CPU mode if running other services:

    ./run_optimisation.sh --directory /media --vmaf 95 --workers 1 --cpu-limit 50

  2. Run during off-peak hours to minimize user impact

  3. Monitor CPU temperature:

    watch -n 2 'sensors | grep "Package id"'

  4. Use higher preset for faster encodes:

    ./run_optimisation.sh --vmaf 93 --preset 8

For Windows PCs (AMD RX 7900 XT)

  1. Enable hardware acceleration for massive speedup:

    .\run_optimisation.ps1 --directory "D:\Movies" --hwaccel auto

  2. Test small sample first to verify settings:

    .\run_optimisation.ps1 --directory "D:\Media\sample" --thorough --vmaf 95

  3. Monitor GPU usage (Task Manager or third-party tools)

  4. Consider quality compensation - GPU encoding may need slightly lower VMAF targets to match CPU quality

For WSL (Ubuntu/Debian)

  1. Access Windows drives via /mnt/c/:

    ls /mnt/c/Media/movies

  2. Increase memory limits if encoding 4K content:

    # Edit ~/.wslconfig
[wsl2]
memory=16GB

Running in Docker (Optional)

# Build image
docker build -t vmaf-optimiser .

# Run with mount
docker run -v /path/to/media:/media vmaf-optimiser /media

Troubleshooting

Issue: Scripts not found

Solution: Ensure you're in the correct directory with the scripts installed.

Issue: "ab-av1: command not found"

Solution: Install ab-av1 via cargo:

cargo install ab-av1

Issue: FFmpeg VMAF not available

Solution: Recompile FFmpeg with VMAF support or download a pre-built version that includes libvmaf.

Issue: Out of Memory

Solution: Reduce workers or increase swap:

# Increase swap
sudo fallocate -l 4G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

# Use more conservative settings
./run_optimisation.sh --workers 1 --vmaf 93

Issue: Multiple machines encoding same file

Solution: This is prevented by lock files. If you see duplicates, check /opt/Optmiser/.lock/ for stale locks.

Issue: Encoding fails

Solution: Check logs:

cat /opt/Optmiser/logs/failed_encodes.jsonl | jq '.'

Issue: "unexpected argument" error

Solution: Use correct flags for your ab-av1 version. The wrapper scripts now validate support at runtime.

Git Workflow

Initial Setup

cd /opt/Optmiser
git init
git remote add origin https://gitea.theflagroup.com/bnair/VMAFOptimiser.git
git branch -M main
git add .
git commit -m "Initial commit: VMAF optimisation pipeline"
git push -u origin main

Daily Updates

cd /opt/Optmiser
git pull origin main

# Run optimisation
./run_optimisation.sh /media tv_movies

# Review changes
git diff

Committing Changes

cd /opt/Optmiser
git status

# Add changed files
git add optimize_library.py run_optimisation.sh run_optimisation.ps1

# Commit with message
git commit -m "feat: add Windows and Linux wrapper scripts"

# Push
git push

Viewing Logs

# Watch logs in real-time
tail -f /opt/Optmiser/logs/tv_movies.jsonl | jq '.'

# Check files logged for review
cat /opt/Optmiser/logs/low_savings_skips.jsonl | jq '.[] | select(.recommendations=="logged_for_review")'

# Statistics
jq -r '.status' /opt/Optmiser/logs/tv_movies.jsonl | sort | uniq -c

# Find what CRF/VMAF combinations are being used most
jq -r '[.vmaf, .crf] | @tsv' /opt/Optmiser/logs/tv_movies.jsonl | sort | uniq -c

View History

cd /opt/Optmiser
git log --oneline
git log --graph --all

FAQ

Q: Can I run this on multiple machines at once? A: Yes! Each machine will process different files due to lock file mechanism.

Q: Should I use Windows or WSL? A: WSL is recommended for Linux compatibility. Use Windows native if you need direct hardware access or performance.

Q: Will hardware encoding work better than CPU? A: For AMD RX 7900 XT, hardware AV1 encoding is ~3-10x faster than CPU. However, GPU encoding may need slightly lower VMAF targets to match CPU quality.

Q: What VMAF target should I use? A: Start with VMAF 94 or 95. Drop to 92-90 if you need more savings.

Q: How do I know which files are being processed? A: Check .lock/ directory: ls -la /opt/Optmiser/.lock/

Q: Can I pause/resume? A: Pause by stopping the script (Ctrl+C). Resume by running again - it skips processed files.

Q: What happens if encoding fails? A: Error is logged to failed_encodes.jsonl. Original file is NOT modified.

Q: How much CPU does encoding use? A: Full CPU by default. Use --workers 1 for single-threaded, or limit with --cpu-limit 50 for 50% (12 threads on 24-core).

Q: What are the log files? A:

  • tv_movies.jsonl - Successful TV & Movie encodes
  • content.jsonl - Successful Content folder encodes
  • low_savings_skips.jsonl - Files with <12% savings + 15% estimates
  • failed_searches.jsonl - Files that couldn't hit any VMAF target
  • failed_encodes.jsonl - Encoding errors

Q: How do I see real-time progress? A: The script streams all ab-av1 output in real-time, showing ETA, encoding speed, and frame count.

Last Updated: December 31, 2025 Version: 2.0 with Windows and Linux Wrapper Scripts

Reference in New Issue View Git Blame Copy Permalink