VMAFOptimiser/SETUP.md

# VMAF Optimisation Pipeline - Setup Guide

## Quick Start

### Prerequisites
- Python 3.8+
- FFmpeg with VMAF support (`ffmpeg -filters 2>&1 | grep libvmaf`)
- ab-av1 binary (v0.10.3+)

### Installation

**On Linux (Server/WSL):**
>>>>>>> 91418fa898de4a73e144a9fb9202b2315e922ab9
```bash
# Clone repository
git clone https://gitea.theflagroup.com/bnair/VMAFOptimiser.git /opt/Optmiser

# Download ab-av1
cd /opt/Optmiser/bin
wget https://github.com/alexheretic/ab-av1/releases/download/v0.10.3/ab-av1-x86_64-unknown-linux-musl
chmod +x ab-av1

# Set up directories
mkdir -p /opt/Optmiser/tmp /opt/Optmiser/logs /opt/Optmiser/.lock
```

**On Windows:**
```powershell
# Clone repository
git clone https://gitea.theflagroup.com/bnair/VMAFOptimiser.git C:\Optmiser

# Install dependencies
winget install ffmpeg

# Install Rust and ab-av1
# Download from https://rust-lang.org/
cargo install ab-av1

# Set up directories
mkdir C:\Optmiser\tmp, C:\Optmiser\logs
```

**On macOS:**
```bash
# Clone repository
git clone https://gitea.theflagroup.com/bnair/VMAFOptimiser.git /opt/Optmiser

# Install dependencies
brew install ffmpeg

# Install Rust and ab-av1
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
cargo install ab-av1

# Set up directories
mkdir -p /opt/Optmiser/tmp /opt/Optmiser/logs /opt/Optmiser/.lock
```

## Running the Optimiser

### Choose Your Script

**Linux / WSL / macOS:**
```bash
# Interactive mode (shows prompts)
./run_optimisation.sh

# Direct execution with parameters
./run_optimisation.sh --directory /mnt/Media/Movies --vmaf 95 --preset 6 --workers 1

# For 50% CPU mode on server
./run_optimisation.sh --directory /mnt/Media/Movies --vmaf 95 --workers 1 --cpu-limit 50
```

**Windows:**
```powershell
# Interactive mode (shows prompts)
.\run_optimisation.ps1

# Direct execution with parameters
.\run_optimisation.ps1 -directory "D:\Movies" --vmaf 95 --preset 6 --workers 1

# For hardware acceleration (AMD GPU)
.\run_optimisation.ps1 -directory "D:\Movies" --vmaf 95 --hwaccel auto
```

## Script Parameters

All wrapper scripts (`run_optimisation.sh` on Linux, `run_optimisation.ps1` on Windows) accept these parameters:

| Parameter | Description | Default |
|------------|-------------|---------|
| `--directory <path>` | Root directory to scan | Current directory |
| `--vmaf <score>` | Target VMAF score | 95.0 |
| `--preset <value>` | SVT-AV1 Preset (4=best, 6=balanced, 8=fast) | 6 |
| `--workers <count>` | Concurrent files to process | 1 |
| `--samples <count>` | Samples for CRF search | 4 |
| `--thorough` | Use thorough mode (slower, more accurate) | false |
| `--encoder <name>` | ab-av1 encoder | svt-av1 |
| `--hwaccel <value>` | Hardware acceleration | none (auto: auto-detect) |

## Multi-Machine Setup

### How Lock Files Prevent Conflicts

Each video file has a lock file: `/opt/Optmiser/.lock/{video_filename}`

**Process:**
1. Machine A: Checks for lock → Not found, creates lock
2. Machine A: Starts encoding
3. Machine B: Checks for lock → Found, skips file
4. Machine A: Finishes, removes lock
5. Machine B: Can now process that file

**Result:** Different machines automatically process different files simultaneously!

### Setup for Multiple Machines

**Machine 1 - Linux Server (Intel i9-12900H):**
```bash
cd /opt/Optmiser
git pull origin main
./run_optimisation.sh /mnt/Media/movies --vmaf 95
```

**Machine 2 - Windows PC (AMD RX 7900 XT):**
```powershell
cd C:\Optmiser
git pull origin main
.\run_optimisation.ps1 D:\Media\movies --vmaf 95 --hwaccel auto
```

**Machine 3 - Another Linux PC:**
```bash
cd /opt/Optmiser
git pull origin main
./run_optimisation.sh /home/user/Media/tv --vmaf 95
```

All three can run simultaneously!

## Hardware Acceleration

### Automatic Detection

When `--hwaccel auto` is specified, the wrapper scripts automatically select the best available hardware acceleration:

| Platform | Auto Selection | Notes |
|-----------|----------------|--------|
| Windows | d3d11va | Direct3D Video Acceleration |
| macOS | videotoolbox | VideoToolbox framework |
| Linux/WSL | vaapi | Video Acceleration via VA-API |

### GPU vs iGPU Priority

- **Discrete GPU takes priority:** If a discrete GPU (like AMD RX 7900 XT) is present, it's selected over integrated GPU
- **For AMD RX 7900 XT:** Hardware encoding provides ~3-10x speedup over CPU
- **Note:** GPU encoding may need slightly lower VMAF targets to match CPU quality

## Customization

### Changing VMAF Targets

Edit `optimize_library.py`:

```python
# More aggressive (smaller files, lower quality)
TARGETS = [92.0, 90.0, 88.0]

# Conservative (larger files, higher quality)
TARGETS = [95.0, 94.0, 93.0]
```

### Changing Savings Threshold

```python
# More aggressive (encode more)
MIN_SAVINGS_PERCENT = 8.0

# Less aggressive (encode fewer)
MIN_SAVINGS_PERCENT = 15.0
```

### Changing Encoder Preset

```python
# Faster encodes (larger files, lower quality)
PRESET = 8

# Better quality (slower encodes, smaller files)
PRESET = 4
```

## Platform-Specific Tips

### For Linux Servers (Intel i9-12900H)

1. **Use 50% CPU mode** if running other services:
   ```bash
   ./run_optimisation.sh --directory /media --vmaf 95 --workers 1 --cpu-limit 50
   ```

2. **Run during off-peak hours** to minimize user impact

3. **Monitor CPU temperature:**
   ```bash
   watch -n 2 'sensors | grep "Package id"'
   ```

4. **Use higher preset for faster encodes:**
   ```bash
   ./run_optimisation.sh --vmaf 93 --preset 8
   ```

### For Windows PCs (AMD RX 7900 XT)

1. **Enable hardware acceleration** for massive speedup:
   ```powershell
   .\run_optimisation.ps1 --directory "D:\Movies" --hwaccel auto
   ```

2. **Test small sample first** to verify settings:
   ```powershell
   .\run_optimisation.ps1 --directory "D:\Media\sample" --thorough --vmaf 95
   ```

3. **Monitor GPU usage** (Task Manager or third-party tools)

4. **Consider quality compensation** - GPU encoding may need slightly lower VMAF targets to match CPU quality

### For WSL (Ubuntu/Debian)

1. **Access Windows drives** via `/mnt/c/`:
   ```bash
   ls /mnt/c/Media/movies
   ```

2. **Increase memory limits** if encoding 4K content:
   ```bash
   # Edit ~/.wslconfig
   [wsl2]
   memory=16GB
   ```

## Running in Docker (Optional)

```bash
# Build image
docker build -t vmaf-optimiser .

# Run with mount
docker run -v /path/to/media:/media vmaf-optimiser /media
```

## Troubleshooting

### Issue: Scripts not found

**Solution:** Ensure you're in the correct directory with the scripts installed.

### Issue: "ab-av1: command not found"

**Solution:** Install ab-av1 via cargo:
```bash
cargo install ab-av1
```

### Issue: FFmpeg VMAF not available

**Solution:** Recompile FFmpeg with VMAF support or download a pre-built version that includes libvmaf.

### Issue: Out of Memory

**Solution:** Reduce workers or increase swap:
```bash
# Increase swap
sudo fallocate -l 4G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

# Use more conservative settings
./run_optimisation.sh --workers 1 --vmaf 93
```

### Issue: Multiple machines encoding same file

**Solution:** This is prevented by lock files. If you see duplicates, check `/opt/Optmiser/.lock/` for stale locks.

### Issue: Encoding fails

**Solution:** Check logs:
```bash
cat /opt/Optmiser/logs/failed_encodes.jsonl | jq '.'
```

### Issue: "unexpected argument" error

**Solution:** Use correct flags for your ab-av1 version. The wrapper scripts now validate support at runtime.

## Git Workflow

### Initial Setup
```bash
cd /opt/Optmiser
git init
git remote add origin https://gitea.theflagroup.com/bnair/VMAFOptimiser.git
git branch -M main
git add .
git commit -m "Initial commit: VMAF optimisation pipeline"
git push -u origin main
```

### Daily Updates
```bash
cd /opt/Optmiser
git pull origin main

# Run optimisation
./run_optimisation.sh /media tv_movies

# Review changes
git diff
```

### Committing Changes

```bash
cd /opt/Optmiser
git status

# Add changed files
git add optimize_library.py run_optimisation.sh run_optimisation.ps1

# Commit with message
git commit -m "feat: add Windows and Linux wrapper scripts"

# Push
git push
```

### Viewing Logs

```bash
# Watch logs in real-time
tail -f /opt/Optmiser/logs/tv_movies.jsonl | jq '.'

# Check files logged for review
cat /opt/Optmiser/logs/low_savings_skips.jsonl | jq '.[] | select(.recommendations=="logged_for_review")'

# Statistics
jq -r '.status' /opt/Optmiser/logs/tv_movies.jsonl | sort | uniq -c

# Find what CRF/VMAF combinations are being used most
jq -r '[.vmaf, .crf] | @tsv' /opt/Optmiser/logs/tv_movies.jsonl | sort | uniq -c
```

### View History

```bash
cd /opt/Optmiser
git log --oneline
git log --graph --all
```

## FAQ

**Q: Can I run this on multiple machines at once?**
A: Yes! Each machine will process different files due to lock file mechanism.

**Q: Should I use Windows or WSL?**
A: WSL is recommended for Linux compatibility. Use Windows native if you need direct hardware access or performance.

**Q: Will hardware encoding work better than CPU?**
A: For AMD RX 7900 XT, hardware AV1 encoding is ~3-10x faster than CPU. However, GPU encoding may need slightly lower VMAF targets to match CPU quality.

**Q: What VMAF target should I use?**
A: Start with VMAF 94 or 95. Drop to 92-90 if you need more savings.

**Q: How do I know which files are being processed?**
A: Check `.lock/` directory: `ls -la /opt/Optmiser/.lock/`

**Q: Can I pause/resume?**
A: Pause by stopping the script (Ctrl+C). Resume by running again - it skips processed files.

**Q: What happens if encoding fails?**
A: Error is logged to `failed_encodes.jsonl`. Original file is NOT modified.

**Q: How much CPU does encoding use?**
A: Full CPU by default. Use `--workers 1` for single-threaded, or limit with `--cpu-limit 50` for 50% (12 threads on 24-core).

**Q: What are the log files?**
A: 
- `tv_movies.jsonl` - Successful TV & Movie encodes
- `content.jsonl` - Successful Content folder encodes
- `low_savings_skips.jsonl` - Files with <12% savings + 15% estimates
- `failed_searches.jsonl` - Files that couldn't hit any VMAF target
- `failed_encodes.jsonl` - Encoding errors

**Q: How do I see real-time progress?**
A: The script streams all ab-av1 output in real-time, showing ETA, encoding speed, and frame count.

---

**Last Updated:** December 31, 2025
**Version:** 2.0 with Windows and Linux Wrapper Scripts