VMAFOptimiser/SETUP.md

# VMAF Optimisation Pipeline - Setup Guide

## Quick Start (Server)

### Prerequisites
- FFmpeg with VMAF support (`ffmpeg -filters 2>&1 | grep libvmaf`)
- Python 3.8+
- ab-av1 binary (v0.10.3+)

### Installation
```bash
# Clone repository
git clone https://gitea.theflagroup.com/bnair/VMAFOptimiser.git /opt/Optmiser

# Download ab-av1 (if not present)
cd /opt/Optmiser/bin
wget https://github.com/alexheretic/ab-av1/releases/download/v0.10.3/ab-av1-x86_64-unknown-linux-musl
chmod +x ab-av1

# Set up temporary directories
mkdir -p /opt/Optmiser/tmp /opt/Optmiser/logs /opt/Optmiser/.lock
```

### Basic Usage
```bash
# Process TV shows
python3 /opt/Optmiser/optimise_media_v2.py /mnt/Media/tv tv_movies

# Process movies
python3 /opt/Optmiser/optimise_media_v2.py /mnt/Media/movies tv_movies

# Process with 50% CPU limit (leaves CPU for other tasks)
python3 /opt/Optmiser/optimise_media_v2.py /mnt/Media/tv tv_movies --cpu-limit 50
```

---

## Quick Start (Windows/WSL)

### WSL Installation (Recommended)
```bash
# Update WSL
wsl --update

# Install dependencies in WSL Ubuntu/Debian
sudo apt update
sudo apt install -y ffmpeg python3 git

# Clone repository into WSL
git clone https://gitea.theflagroup.com/bnair/VMAFOptimiser.git /opt/Optmiser

# Install ab-av1
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source ~/.cargo/env
cargo install ab-av1

# Access Windows media from WSL
# Windows C:\ is at /mnt/c/
python3 /opt/Optmiser/optimise_media_v2.py /mnt/c/Media/tv tv_movies
```

### Native Windows Installation
```powershell
# Install Python (if not present)
# Download from python.org

# Install FFmpeg
winget install ffmpeg

# Install Rust
# Download from rust-lang.org

# Install ab-av1
cargo install ab-av1

# Clone repository
git clone https://gitea.theflagroup.com/bnair/VMAFOptimiser.git C:\Optmiser

# Run
python C:\Optmiser\optimise_media_v2.py D:\Media\tv tv_movies
```

---

## Running on Multiple Machines

### How Lock Files Work

Each video file has a corresponding lock file:
```
/opt/Optmiser/.lock/{video_filename}.lock
```

**Process:**
1. Machine A checks for lock → None found, creates lock
2. Machine A starts encoding
3. Machine B checks for lock → Found, skips file
4. Machine A finishes, removes lock
5. Machine B can now process that file

**Result:** Different machines automatically process different files!

### Setup for Multiple Machines

**Machine 1 - Remote Server (Intel i9-12900H):**
```bash
cd /opt/Optmiser
git pull origin main

# Run on /mnt/Media (Linux filesystem)
python3 optimise_media_v2.py /mnt/Media/tv tv_movies
python3 optimise_media_v2.py /mnt/Media/movies tv_movies

# With 50% CPU limit (recommended)
python3 optimise_media_v2.py /mnt/Media/tv tv_movies --cpu-limit 50
```

**Machine 2 - Local PC (AMD RX 7900 XT, Windows):**

Option A - Native Windows:
```powershell
# Map network drive if needed
# \\Server\media\ or use local storage

cd C:\Optmiser
git pull origin main

# Run on local media
python optimise_media_v2.py D:\Media\tv tv_movies
```

Option B - WSL (Recommended):
```bash
# Windows C: drive accessible at /mnt/c/
cd /opt/Optmiser
python optimise_media_v2.py /mnt/c/Media/tv tv_movies
```

**Machine 3 - Another PC (AMD 9800X3D, Linux):**
```bash
cd /opt/Optmiser
git pull origin main

# Run on local media directory
python optimise_media_v2.py /home/user/Media/tv tv_movies
```

All three can run simultaneously - lock files prevent duplicates!

---

## Hardware Encoding Guide

### Detecting Hardware

The script automatically checks for:
1. **AMD GPU encoding support** (future feature)
2. **System thread count**
3. **Operating system** (Linux vs Windows)

### CPU Encoding (Software)

**Best for:** Servers, multi-tasking

```bash
# Force CPU encoding
python3 optimise_media_v2.py /media --use-cpu

# Limit to 50% of CPU (12 threads on 24-core)
python3 optimise_media_v2.py /media --cpu-limit 50

# Set specific worker count
python3 optimise_media_v2.py /media --workers 8
```

**When to use:**
- Running Plex/Jellyfin on same machine
- Server environment with multiple users
- Want to leave GPU free for other tasks

**Expected Speed:** 8-15 fps @ 1080p

### GPU Encoding (AMD RX 7900 XT)

**Status:** Planned for future versions
**Expected Speedup:** 3-10x vs CPU encoding

```bash
# Enable hardware encoding (when implemented)
python3 optimise_media_v2.py /media --use-hardware
```

**When to use:**
- Local PC with AMD GPU
- Want faster encoding speeds
- Can compensate quality with slightly lower VMAF target

**Expected Speed:** 150+ fps @ 1080p

### Quality Compensation for GPU Encoding

GPU encoding may need quality adjustments:

| CPU VMAF | Equivalent GPU VMAF | Why |
|------------|----------------------|-----|
| 94 | 92-93 | GPU has quality limitations |
| 93 | 91-92 | Hardware encoding trade-offs |
| 90 | 88-89 | Significant compression |

**Recommendation:** If using GPU encoding, set `TARGETS = [92.0, 90.0, 88.0]` for similar quality.

---

## Windows/WSL Path Mapping

### Understanding /mnt/c/

In WSL, Windows drives are mapped:

| Windows | WSL Path | Example |
|---------|------------|---------|
| C:\ | /mnt/c/ | /mnt/c/Users/bnair/Downloads |
| D:\ | /mnt/d/ | /mnt/d/Movies/ |
| E:\ | /mnt/e/ | /mnt/e/TV\ |

**To access Windows media from WSL:**
```bash
# List C:\Media\tv
ls /mnt/c/Media/tv

# Process from WSL
python3 /opt/Optmiser/optimise_media_v2.py /mnt/c/Media/tv tv_movies
```

### Network Drives on WSL

```bash
# Map network drive (one-time)
sudo mkdir -p /mnt/server
echo "//192.168.1.100/media | /mnt/server cifs credentials=/path/to/credfile,uid=1000,gid=1000 0 0" | sudo tee -a /etc/fstab

# Access network media
python3 /opt/Optmiser/optimise_media_v2.py /mnt/server/Media/tv tv_movies
```

---

## Customization

### Changing VMAF Targets

Edit `optimise_media_v2.py`:

```python
# Line 15
TARGETS = [94.0, 93.0, 92.0, 90.0]

# More aggressive (smaller files, lower quality)
TARGETS = [92.0, 90.0, 88.0]

# Conservative (larger files, higher quality)
TARGETS = [95.0, 94.0, 93.0]
```

### Changing Savings Threshold

```python
# Line 17
MIN_SAVINGS_PERCENT = 12.0  # Current threshold
MIN_SAVINGS_PERCENT = 8.0   # More aggressive (encode more)
MIN_SAVINGS_PERCENT = 15.0  # Less aggressive (encode fewer)
```

### Changing Encoder Preset

```python
# Line 19
PRESET = 6

# Faster encodes (larger files, lower quality)
PRESET = 8

# Better quality (slower encodes, smaller files)
PRESET = 4
```

### Changing Estimate Target

```python
# Line 18
TARGET_SAVINGS_FOR_ESTIMATE = 15.0

# Target higher savings
TARGET_SAVINGS_FOR_ESTIMATE = 20.0
```

---

## Monitoring

### Watch Progress in Real-Time

```bash
# Tail log file with JSON formatting
tail -f /opt/Optmiser/logs/tv_movies.jsonl | jq '.'

# Monitor encoding speed
watch -n 5 'jq -r "[.savings, .duration]" /opt/Optmiser/logs/tv_movies.jsonl | tail -1'

# Check lock files (what's being processed)
ls -la /opt/Optmiser/.lock/
```

### Performance Dashboard

```bash
# Create a simple dashboard
watch -n 10 '
echo "=== VMAF Optimiser Status ==="
echo ""
echo "Recent Encodes:"
tail -3 /opt/Optmiser/logs/tv_movies.jsonl | jq -r "[.file, .savings, .duration] | @tsv"
echo ""
echo "CPU Usage:"
top -bn1 | head -5
'
```

---

## Troubleshooting

### Issue: Scripts not found

```bash
# Check path
ls /opt/Optmiser/optimise_media_v2.py

# Check Python version
python3 --version  # Should be 3.8+

# Check ab-av1
/opt/Optmiser/bin/ab-av1 --version  # Should be 0.10.3+
```

### Issue: Permission denied

```bash
# Make scripts executable
chmod +x /opt/Optmiser/*.py
chmod +x /opt/Optmiser/*.sh

# Fix lock directory permissions
chmod 777 /opt/Optmiser/.lock
```

### Issue: ab-av1 not found

```bash
# Check if in PATH
which ab-av1

# Use full path if not in PATH
/opt/Optmiser/bin/ab-av1 --version

# Add to PATH
export PATH="$PATH:/opt/Optmiser/bin"
```

### Issue: FFmpeg VMAF not available

```bash
# Check libvmaf
ffmpeg -filters 2>&1 | grep libvmaf

# If not found, recompile FFmpeg with VMAF
# Or download compiled version from johnvansickle.com/ffmpeg
```

### Issue: Out of Memory

```bash
# Check available memory
free -h

# Reduce workers
python3 optimise_media_v2.py /media --workers 4

# Increase swap (if needed)
sudo fallocate -l 4G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
```

---

## Git Workflow

### Initial Setup
```bash
cd /opt/Optmiser
git init
git remote add origin https://gitea.theflagroup.com/bnair/VMAFOptimiser.git
git branch -M main
git add .
git commit -m "Initial commit: VMAF optimisation pipeline"
git push -u origin main
```

### Daily Updates
```bash
cd /opt/Optmiser
git pull origin main

# Run optimisation
python3 optimise_media_v2.py /media tv_movies

# Review changes
git diff
```

### Commit Changes
```bash
cd /opt/Optmiser
git status

# Add changed files
git add optimise_media_v2.py run_optimisation.sh

# Commit with message
git commit -m "feat: add X"

# Push
git push
```

### View History
```bash
cd /opt/Optmiser
git log --oneline
git log --graph --all
```

---

## FAQ

**Q: Can I run this on multiple machines at once?**
A: Yes! Each machine will process different files due to lock file mechanism.

**Q: Will hardware encoding be available?**
A: Planned for future versions. Currently uses CPU encoding (software AV1).

**Q: How do I know if a file is being encoded elsewhere?**
A: Check `/opt/Optmiser/.lock/` - if a lock exists, another machine is processing it.

**Q: Can I change the VMAF target?**
A: Yes, edit `TARGETS = [94.0, 93.0, 92.0, 90.0]` in `optimise_media_v2.py`.

**Q: What if savings are always below 12%?**
A: The script will log 15% estimates to `low_savings_skips.jsonl`. Review these logs to decide if encoding is worth it.

**Q: Does this work on Windows/WSL?**
A: Yes! See the Windows/WSL section for setup instructions.

**Q: How much CPU does encoding use?**
A: Full CPU (24 threads) by default. Use `--cpu-limit 50` for 50% mode.

**Q: Can I pause/resume?**
A: Pause by stopping the script (Ctrl+C). Resume by running again - it skips processed files.

**Q: What happens if encoding fails?**
A: Error is logged to `failed_encodes.jsonl` with error code. Original file is NOT modified.

---

## Support

For issues or questions:
- Check AGENTS.md for detailed technical documentation
- Review logs in `/opt/Optmiser/logs/`
- Test with `--dry-run` flag first

---

**Last Updated:** December 31, 2025