bnair/VMAFOptimiser
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
b8f6c60b5efb3398b6e76fd71d78b478236dac70
VMAFOptimiser/SETUP.md
bnair123 b8f6c60b5e Initial commit: VMAF optimisation pipeline
2025-12-31 17:35:03 +04:00

10 KiB
Raw Blame History

VMAF Optimisation Pipeline - Setup Guide

Quick Start (Server)

Prerequisites

  • FFmpeg with VMAF support (ffmpeg -filters 2>&1 | grep libvmaf)
  • Python 3.8+
  • ab-av1 binary (v0.10.3+)

Installation

# Clone repository
git clone https://gitea.theflagroup.com/bnair/VMAFOptimiser.git /opt/Optmiser

# Download ab-av1 (if not present)
cd /opt/Optmiser/bin
wget https://github.com/alexheretic/ab-av1/releases/download/v0.10.3/ab-av1-x86_64-unknown-linux-musl
chmod +x ab-av1

# Set up temporary directories
mkdir -p /opt/Optmiser/tmp /opt/Optmiser/logs /opt/Optmiser/.lock

Basic Usage

# Process TV shows
python3 /opt/Optmiser/optimise_media_v2.py /mnt/Media/tv tv_movies

# Process movies
python3 /opt/Optmiser/optimise_media_v2.py /mnt/Media/movies tv_movies

# Process with 50% CPU limit (leaves CPU for other tasks)
python3 /opt/Optmiser/optimise_media_v2.py /mnt/Media/tv tv_movies --cpu-limit 50

Quick Start (Windows/WSL)

WSL Installation (Recommended)

# Update WSL
wsl --update

# Install dependencies in WSL Ubuntu/Debian
sudo apt update
sudo apt install -y ffmpeg python3 git

# Clone repository into WSL
git clone https://gitea.theflagroup.com/bnair/VMAFOptimiser.git /opt/Optmiser

# Install ab-av1
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source ~/.cargo/env
cargo install ab-av1

# Access Windows media from WSL
# Windows C:\ is at /mnt/c/
python3 /opt/Optmiser/optimise_media_v2.py /mnt/c/Media/tv tv_movies

Native Windows Installation

# Install Python (if not present)
# Download from python.org

# Install FFmpeg
winget install ffmpeg

# Install Rust
# Download from rust-lang.org

# Install ab-av1
cargo install ab-av1

# Clone repository
git clone https://gitea.theflagroup.com/bnair/VMAFOptimiser.git C:\Optmiser

# Run
python C:\Optmiser\optimise_media_v2.py D:\Media\tv tv_movies

Running on Multiple Machines

How Lock Files Work

Each video file has a corresponding lock file:

/opt/Optmiser/.lock/{video_filename}.lock

Process:

  1. Machine A checks for lock → None found, creates lock
  2. Machine A starts encoding
  3. Machine B checks for lock → Found, skips file
  4. Machine A finishes, removes lock
  5. Machine B can now process that file

Result: Different machines automatically process different files!

Setup for Multiple Machines

Machine 1 - Remote Server (Intel i9-12900H):

cd /opt/Optmiser
git pull origin main

# Run on /mnt/Media (Linux filesystem)
python3 optimise_media_v2.py /mnt/Media/tv tv_movies
python3 optimise_media_v2.py /mnt/Media/movies tv_movies

# With 50% CPU limit (recommended)
python3 optimise_media_v2.py /mnt/Media/tv tv_movies --cpu-limit 50

Machine 2 - Local PC (AMD RX 7900 XT, Windows):

Option A - Native Windows:

# Map network drive if needed
# \\Server\media\ or use local storage

cd C:\Optmiser
git pull origin main

# Run on local media
python optimise_media_v2.py D:\Media\tv tv_movies

Option B - WSL (Recommended):

# Windows C: drive accessible at /mnt/c/
cd /opt/Optmiser
python optimise_media_v2.py /mnt/c/Media/tv tv_movies

Machine 3 - Another PC (AMD 9800X3D, Linux):

cd /opt/Optmiser
git pull origin main

# Run on local media directory
python optimise_media_v2.py /home/user/Media/tv tv_movies

All three can run simultaneously - lock files prevent duplicates!

Hardware Encoding Guide

Detecting Hardware

The script automatically checks for:

  1. AMD GPU encoding support (future feature)
  2. System thread count
  3. Operating system (Linux vs Windows)

CPU Encoding (Software)

Best for: Servers, multi-tasking

# Force CPU encoding
python3 optimise_media_v2.py /media --use-cpu

# Limit to 50% of CPU (12 threads on 24-core)
python3 optimise_media_v2.py /media --cpu-limit 50

# Set specific worker count
python3 optimise_media_v2.py /media --workers 8

When to use:

  • Running Plex/Jellyfin on same machine
  • Server environment with multiple users
  • Want to leave GPU free for other tasks

Expected Speed: 8-15 fps @ 1080p

GPU Encoding (AMD RX 7900 XT)

Status: Planned for future versions Expected Speedup: 3-10x vs CPU encoding

# Enable hardware encoding (when implemented)
python3 optimise_media_v2.py /media --use-hardware

When to use:

  • Local PC with AMD GPU
  • Want faster encoding speeds
  • Can compensate quality with slightly lower VMAF target

Expected Speed: 150+ fps @ 1080p

Quality Compensation for GPU Encoding

GPU encoding may need quality adjustments:

CPU VMAF Equivalent GPU VMAF Why
94 92-93 GPU has quality limitations
93 91-92 Hardware encoding trade-offs
90 88-89 Significant compression

Recommendation: If using GPU encoding, set TARGETS = [92.0, 90.0, 88.0] for similar quality.

Windows/WSL Path Mapping

Understanding /mnt/c/

In WSL, Windows drives are mapped:

Windows WSL Path Example
C:\ /mnt/c/ /mnt/c/Users/bnair/Downloads
D:\ /mnt/d/ /mnt/d/Movies/
E:\ /mnt/e/ /mnt/e/TV\

To access Windows media from WSL:

# List C:\Media\tv
ls /mnt/c/Media/tv

# Process from WSL
python3 /opt/Optmiser/optimise_media_v2.py /mnt/c/Media/tv tv_movies

Network Drives on WSL

# Map network drive (one-time)
sudo mkdir -p /mnt/server
echo "//192.168.1.100/media | /mnt/server cifs credentials=/path/to/credfile,uid=1000,gid=1000 0 0" | sudo tee -a /etc/fstab

# Access network media
python3 /opt/Optmiser/optimise_media_v2.py /mnt/server/Media/tv tv_movies

Customization

Changing VMAF Targets

Edit optimise_media_v2.py:

# Line 15
TARGETS = [94.0, 93.0, 92.0, 90.0]

# More aggressive (smaller files, lower quality)
TARGETS = [92.0, 90.0, 88.0]

# Conservative (larger files, higher quality)
TARGETS = [95.0, 94.0, 93.0]

Changing Savings Threshold

# Line 17
MIN_SAVINGS_PERCENT = 12.0  # Current threshold
MIN_SAVINGS_PERCENT = 8.0   # More aggressive (encode more)
MIN_SAVINGS_PERCENT = 15.0  # Less aggressive (encode fewer)

Changing Encoder Preset

# Line 19
PRESET = 6

# Faster encodes (larger files, lower quality)
PRESET = 8

# Better quality (slower encodes, smaller files)
PRESET = 4

Changing Estimate Target

# Line 18
TARGET_SAVINGS_FOR_ESTIMATE = 15.0

# Target higher savings
TARGET_SAVINGS_FOR_ESTIMATE = 20.0

Monitoring

Watch Progress in Real-Time

# Tail log file with JSON formatting
tail -f /opt/Optmiser/logs/tv_movies.jsonl | jq '.'

# Monitor encoding speed
watch -n 5 'jq -r "[.savings, .duration]" /opt/Optmiser/logs/tv_movies.jsonl | tail -1'

# Check lock files (what's being processed)
ls -la /opt/Optmiser/.lock/

Performance Dashboard

# Create a simple dashboard
watch -n 10 '
echo "=== VMAF Optimiser Status ==="
echo ""
echo "Recent Encodes:"
tail -3 /opt/Optmiser/logs/tv_movies.jsonl | jq -r "[.file, .savings, .duration] | @tsv"
echo ""
echo "CPU Usage:"
top -bn1 | head -5
'

Troubleshooting

Issue: Scripts not found

# Check path
ls /opt/Optmiser/optimise_media_v2.py

# Check Python version
python3 --version  # Should be 3.8+

# Check ab-av1
/opt/Optmiser/bin/ab-av1 --version  # Should be 0.10.3+

Issue: Permission denied

# Make scripts executable
chmod +x /opt/Optmiser/*.py
chmod +x /opt/Optmiser/*.sh

# Fix lock directory permissions
chmod 777 /opt/Optmiser/.lock

Issue: ab-av1 not found

# Check if in PATH
which ab-av1

# Use full path if not in PATH
/opt/Optmiser/bin/ab-av1 --version

# Add to PATH
export PATH="$PATH:/opt/Optmiser/bin"

Issue: FFmpeg VMAF not available

# Check libvmaf
ffmpeg -filters 2>&1 | grep libvmaf

# If not found, recompile FFmpeg with VMAF
# Or download compiled version from johnvansickle.com/ffmpeg

Issue: Out of Memory

# Check available memory
free -h

# Reduce workers
python3 optimise_media_v2.py /media --workers 4

# Increase swap (if needed)
sudo fallocate -l 4G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

Git Workflow

Initial Setup

cd /opt/Optmiser
git init
git remote add origin https://gitea.theflagroup.com/bnair/VMAFOptimiser.git
git branch -M main
git add .
git commit -m "Initial commit: VMAF optimisation pipeline"
git push -u origin main

Daily Updates

cd /opt/Optmiser
git pull origin main

# Run optimisation
python3 optimise_media_v2.py /media tv_movies

# Review changes
git diff

Commit Changes

cd /opt/Optmiser
git status

# Add changed files
git add optimise_media_v2.py run_optimisation.sh

# Commit with message
git commit -m "feat: add X"

# Push
git push

View History

cd /opt/Optmiser
git log --oneline
git log --graph --all

FAQ

Q: Can I run this on multiple machines at once? A: Yes! Each machine will process different files due to lock file mechanism.

Q: Will hardware encoding be available? A: Planned for future versions. Currently uses CPU encoding (software AV1).

Q: How do I know if a file is being encoded elsewhere? A: Check /opt/Optmiser/.lock/ - if a lock exists, another machine is processing it.

Q: Can I change the VMAF target? A: Yes, edit TARGETS = [94.0, 93.0, 92.0, 90.0] in optimise_media_v2.py.

Q: What if savings are always below 12%? A: The script will log 15% estimates to low_savings_skips.jsonl. Review these logs to decide if encoding is worth it.

Q: Does this work on Windows/WSL? A: Yes! See the Windows/WSL section for setup instructions.

Q: How much CPU does encoding use? A: Full CPU (24 threads) by default. Use --cpu-limit 50 for 50% mode.

Q: Can I pause/resume? A: Pause by stopping the script (Ctrl+C). Resume by running again - it skips processed files.

Q: What happens if encoding fails? A: Error is logged to failed_encodes.jsonl with error code. Original file is NOT modified.

Support

For issues or questions:

  • Check AGENTS.md for detailed technical documentation
  • Review logs in /opt/Optmiser/logs/
  • Test with --dry-run flag first

Last Updated: December 31, 2025

Reference in New Issue View Git Blame Copy Permalink