bnair/VMAFOptimiser
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
ef81365fce54137caacaf13672c5335d3c5788d0
VMAFOptimiser/SETUP.md
bnair123 b4a82e0db5 Added actualy HW encode
2025-12-31 23:39:21 +04:00

13 KiB
Raw Blame History

VMAF Optimisation Pipeline - Setup Guide

Quick Start

Prerequisites

  • Python 3.8+
  • FFmpeg with VMAF support (ffmpeg -filters 2>&1 | grep libvmaf)
  • ab-av1 binary (v0.10.3+)

Installation

On Linux (Server/WSL):

# Clone repository
git clone https://gitea.theflagroup.com/bnair/VMAFOptimiser.git /opt/Optmiser

# Download ab-av1
cd /opt/Optmiser/bin
wget https://github.com/alexheretic/ab-av1/releases/download/v0.10.3/ab-av1-x86_64-unknown-linux-musl
chmod +x ab-av1

# Set up directories
mkdir -p /opt/Optmiser/tmp /opt/Optmiser/logs /opt/Optmiser/.lock

On Windows:

# Clone repository
git clone https://gitea.theflagroup.com/bnair/VMAFOptimiser.git C:\Optmiser

# Install dependencies
winget install ffmpeg

# Install Rust and ab-av1
# Download from https://rust-lang.org/
cargo install ab-av1

# Set up directories
mkdir C:\Optmiser\tmp, C:\Optmiser\logs

On macOS:

# Clone repository
git clone https://gitea.theflagroup.com/bnair/VMAFOptimiser.git /opt/Optmiser

# Install dependencies
brew install ffmpeg

# Install Rust and ab-av1
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
cargo install ab-av1

# Set up directories
mkdir -p /opt/Optmiser/tmp /opt/Optmiser/logs /opt/Optmiser/.lock

Running the Optimiser

Choose Your Script

Linux / WSL / macOS:

# Interactive mode (shows prompts)
./run_optimisation.sh

# Direct execution with parameters
./run_optimisation.sh --directory /mnt/Media/Movies --vmaf 95 --preset 6 --workers 1

# For 50% CPU mode on server
./run_optimisation.sh --directory /mnt/Media/Movies --vmaf 95 --workers 1 --cpu-limit 50

Windows:

# Interactive mode (shows prompts)
.\run_optimisation.ps1

# Direct execution with parameters
.\run_optimisation.ps1 -directory "D:\Movies" --vmaf 95 --preset 6 --workers 1

# For hardware acceleration (AMD GPU)
.\run_optimisation.ps1 -directory "D:\Movies" --vmaf 95 --hwaccel auto

Script Parameters

All wrapper scripts (run_optimisation.sh on Linux, run_optimisation.ps1 on Windows) accept these parameters:

Parameter Description Default
--directory <path> Root directory to scan Current directory
--vmaf <score> Target VMAF score 95.0
--preset <value> SVT-AV1 Preset (4=best, 6=balanced, 8=fast) 6
--workers <count> Concurrent files to process 1
--samples <count> Samples for CRF search 4
--encoder <name> Video encoder: svt-av1, av1_amf, av1_nvenc, av1_qsv svt-av1
--hwaccel <value> Hardware decode acceleration (auto, d3d11va, vaapi) none
--use-hardware-worker Use 1 HW encoder worker + rest CPU workers false
--plex-url <url> Plex server URL for library refresh none
--plex-token <token> Plex auth token none
--log-dir <path> Log directory /opt/Optmiser/logs

Multi-Machine Setup

How Lock Files Prevent Conflicts

Each video file has a lock file: /opt/Optmiser/.lock/{video_filename}

Process:

  1. Machine A: Checks for lock → Not found, creates lock
  2. Machine A: Starts encoding
  3. Machine B: Checks for lock → Found, skips file
  4. Machine A: Finishes, removes lock
  5. Machine B: Can now process that file

Result: Different machines automatically process different files simultaneously!

Setup for Multiple Machines

Machine 1 - Linux Server (Intel i9-12900H):

cd /opt/Optmiser
git pull origin main
./run_optimisation.sh /mnt/Media/movies --vmaf 95

Machine 2 - Windows PC (AMD RX 7900 XT):

cd C:\Optmiser
git pull origin main
.\run_optimisation.ps1 D:\Media\movies --vmaf 95 --hwaccel auto

Machine 3 - Another Linux PC:

cd /opt/Optmiser
git pull origin main
./run_optimisation.sh /home/user/Media/tv --vmaf 95

All three can run simultaneously!

Hardware Acceleration

Hardware Decoding vs Hardware Encoding

There are two types of hardware acceleration:

  1. Hardware Decoding (--hwaccel): Uses GPU to decode source video faster. Doesn't affect quality.
  2. Hardware Encoding (--encoder): Uses GPU to encode output video. Much faster but requires quality compensation.

Hardware Encoders

Encoder GPU Platform Notes
svt-av1 CPU All Default. Best quality, slowest.
av1_amf AMD Windows/Linux RX 7000 series, ~3-10x faster than CPU
av1_nvenc NVIDIA Windows/Linux RTX 40 series, very fast
av1_qsv Intel Windows/Linux Arc GPUs, Intel iGPU with AV1

FFmpeg with Hardware Encoder Support

Windows (pre-built with HW encoders):

# Most Windows FFmpeg builds include hardware encoders
# Verify with:
ffmpeg -encoders 2>&1 | findstr av1

# Should show: av1_amf, av1_nvenc, av1_qsv (depending on your GPU)

# If missing, download from: https://github.com/BtbN/FFmpeg-Builds/releases
# Choose: ffmpeg-master-latest-win64-gpl.zip (includes all encoders)

Linux (may need custom build):

# Check available encoders
ffmpeg -encoders 2>&1 | grep av1

# For AMD (av1_amf) - requires AMF SDK
# Install AMD GPU drivers with AMF support
sudo apt install amdgpu-pro

# For NVIDIA (av1_nvenc) - requires NVIDIA drivers
sudo apt install nvidia-driver-535  # or newer

# For Intel (av1_qsv) - requires Intel Media SDK
sudo apt install intel-media-va-driver-non-free

# If your distro's ffmpeg lacks HW encoders, use static build:
wget https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz

Using Hardware Workers (1 GPU + 3 CPU)

For mixed encoding with your RX 7900 XT:

# Windows - 4 workers: 1 uses GPU (av1_amf), 3 use CPU (svt-av1)
.\run_optimisation.ps1 `
    -Directory "Z:\" `
    -Vmaf 94 `
    -Workers 4 `
    -Encoder av1_amf `
    -Hwaccel auto `
    -UseHardwareWorker `
    -LogDir "C:\Users\bnair\Documents\VMAFOptimiser\logs"

# Linux - same concept
./run_optimisation.sh \
    --directory /media \
    --vmaf 94 \
    --workers 4 \
    --encoder av1_amf \
    --hwaccel auto \
    --use-hardware-worker

How it works:

  • First worker to start claims the GPU and uses av1_amf
  • Remaining 3 workers use svt-av1 (CPU)
  • GPU worker applies +2 VMAF offset automatically to match CPU quality

Quality Compensation

Hardware encoders produce lower quality at the same settings. The script automatically compensates:

Target VMAF CPU (svt-av1) GPU (av1_amf)
94 Searches for VMAF 94 Searches for VMAF 96
93 Searches for VMAF 93 Searches for VMAF 95
92 Searches for VMAF 92 Searches for VMAF 94

This offset (HW_ENCODER_VMAF_OFFSET = 2.0) can be adjusted in optimize_library.py.

Automatic Detection

When --hwaccel auto is specified, the wrapper scripts automatically select the best available hardware acceleration:

Platform Auto Selection Notes
Windows d3d11va Direct3D Video Acceleration
macOS videotoolbox VideoToolbox framework
Linux/WSL vaapi Video Acceleration via VA-API

GPU vs iGPU Priority

  • Discrete GPU takes priority: If a discrete GPU (like AMD RX 7900 XT) is present, it's selected over integrated GPU
  • For AMD RX 7900 XT: Hardware encoding provides ~3-10x speedup over CPU
  • Note: GPU encoding may need slightly lower VMAF targets to match CPU quality

Customization

Changing VMAF Targets

Edit optimize_library.py:

# More aggressive (smaller files, lower quality)
TARGETS = [92.0, 90.0, 88.0]

# Conservative (larger files, higher quality)
TARGETS = [95.0, 94.0, 93.0]

Changing Savings Threshold

# More aggressive (encode more)
MIN_SAVINGS_PERCENT = 8.0

# Less aggressive (encode fewer)
MIN_SAVINGS_PERCENT = 15.0

Changing Encoder Preset

# Faster encodes (larger files, lower quality)
PRESET = 8

# Better quality (slower encodes, smaller files)
PRESET = 4

Platform-Specific Tips

For Linux Servers (Intel i9-12900H)

  1. Use 50% CPU mode if running other services:

    ./run_optimisation.sh --directory /media --vmaf 95 --workers 1 --cpu-limit 50

  2. Run during off-peak hours to minimize user impact

  3. Monitor CPU temperature:

    watch -n 2 'sensors | grep "Package id"'

  4. Use higher preset for faster encodes:

    ./run_optimisation.sh --vmaf 93 --preset 8

For Windows PCs (AMD RX 7900 XT)

  1. Enable hardware acceleration for massive speedup:

    .\run_optimisation.ps1 --directory "D:\Movies" --hwaccel auto

  2. Test small sample first to verify settings:

    .\run_optimisation.ps1 --directory "D:\Media\sample" --thorough --vmaf 95

  3. Monitor GPU usage (Task Manager or third-party tools)

  4. Consider quality compensation - GPU encoding may need slightly lower VMAF targets to match CPU quality

For WSL (Ubuntu/Debian)

  1. Access Windows drives via /mnt/c/:

    ls /mnt/c/Media/movies

  2. Increase memory limits if encoding 4K content:

    # Edit ~/.wslconfig
[wsl2]
memory=16GB

Running in Docker (Optional)

# Build image
docker build -t vmaf-optimiser .

# Run with mount
docker run -v /path/to/media:/media vmaf-optimiser /media

Troubleshooting

Issue: Scripts not found

Solution: Ensure you're in the correct directory with the scripts installed.

Issue: "ab-av1: command not found"

Solution: Install ab-av1 via cargo:

cargo install ab-av1

Issue: FFmpeg VMAF not available

Solution: Recompile FFmpeg with VMAF support or download a pre-built version that includes libvmaf.

Issue: Out of Memory

Solution: Reduce workers or increase swap:

# Increase swap
sudo fallocate -l 4G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

# Use more conservative settings
./run_optimisation.sh --workers 1 --vmaf 93

Issue: Multiple machines encoding same file

Solution: This is prevented by lock files. If you see duplicates, check /opt/Optmiser/.lock/ for stale locks.

Issue: Encoding fails

Solution: Check logs:

cat /opt/Optmiser/logs/failed_encodes.jsonl | jq '.'

Issue: "unexpected argument" error

Solution: Use correct flags for your ab-av1 version. The wrapper scripts now validate support at runtime.

Git Workflow

Initial Setup

cd /opt/Optmiser
git init
git remote add origin https://gitea.theflagroup.com/bnair/VMAFOptimiser.git
git branch -M main
git add .
git commit -m "Initial commit: VMAF optimisation pipeline"
git push -u origin main

Daily Updates

cd /opt/Optmiser
git pull origin main

# Run optimisation
./run_optimisation.sh /media tv_movies

# Review changes
git diff

Committing Changes

cd /opt/Optmiser
git status

# Add changed files
git add optimize_library.py run_optimisation.sh run_optimisation.ps1

# Commit with message
git commit -m "feat: add Windows and Linux wrapper scripts"

# Push
git push

Viewing Logs

# Watch logs in real-time
tail -f /opt/Optmiser/logs/tv_movies.jsonl | jq '.'

# Check files logged for review
cat /opt/Optmiser/logs/low_savings_skips.jsonl | jq '.[] | select(.recommendations=="logged_for_review")'

# Statistics
jq -r '.status' /opt/Optmiser/logs/tv_movies.jsonl | sort | uniq -c

# Find what CRF/VMAF combinations are being used most
jq -r '[.vmaf, .crf] | @tsv' /opt/Optmiser/logs/tv_movies.jsonl | sort | uniq -c

View History

cd /opt/Optmiser
git log --oneline
git log --graph --all

FAQ

Q: Can I run this on multiple machines at once? A: Yes! Each machine will process different files due to lock file mechanism.

Q: Should I use Windows or WSL? A: WSL is recommended for Linux compatibility. Use Windows native if you need direct hardware access or performance.

Q: Will hardware encoding work better than CPU? A: For AMD RX 7900 XT, hardware AV1 encoding is ~3-10x faster than CPU. However, GPU encoding may need slightly lower VMAF targets to match CPU quality.

Q: What VMAF target should I use? A: Start with VMAF 94 or 95. Drop to 92-90 if you need more savings.

Q: How do I know which files are being processed? A: Check .lock/ directory: ls -la /opt/Optmiser/.lock/

Q: Can I pause/resume? A: Pause by stopping the script (Ctrl+C). Resume by running again - it skips processed files.

Q: What happens if encoding fails? A: Error is logged to failed_encodes.jsonl. Original file is NOT modified.

Q: How much CPU does encoding use? A: Full CPU by default. Use --workers 1 for single-threaded, or limit with --cpu-limit 50 for 50% (12 threads on 24-core).

Q: What are the log files? A:

  • tv_movies.jsonl - Successful TV & Movie encodes
  • content.jsonl - Successful Content folder encodes
  • low_savings_skips.jsonl - Files with <12% savings + 15% estimates
  • failed_searches.jsonl - Files that couldn't hit any VMAF target
  • failed_encodes.jsonl - Encoding errors

Q: How do I see real-time progress? A: The script streams all ab-av1 output in real-time, showing ETA, encoding speed, and frame count.

Last Updated: December 31, 2025 Version: 2.0 with Windows and Linux Wrapper Scripts

Reference in New Issue View Git Blame Copy Permalink