# Running Notebooks in Background

## Quick: Check Ray Tune Progress

```bash
# Is papermill still running?
ps aux | grep papermill | grep -v grep

# View live log
tail -f papermill.log

# Find latest Ray Tune run and count completed trials
LATEST=$(ls -td ~/ray_results/trainable_* 2>/dev/null | head -1)
echo "Run: $LATEST"
COMPLETED=$(find "$LATEST" -name "result.json" -size +0 2>/dev/null | wc -l)
TOTAL=$(ls -d "$LATEST"/trainable_*/ 2>/dev/null | wc -l)
echo "Completed: $COMPLETED / $TOTAL"

# Check workers are healthy
for port in 8001 8002 8003; do
  status=$(curl -s "localhost:$port/health" 2>/dev/null | python3 -c "import sys,json; print(json.load(sys.stdin).get('status','down'))" 2>/dev/null || echo "down")
  echo "Worker $port: $status"
done

# Show best result so far
if [ "$COMPLETED" -gt 0 ]; then
  find "$LATEST" -name "result.json" -size +0 -exec cat {} \; 2>/dev/null | \
    python3 -c "import sys,json; results=[json.loads(l) for l in sys.stdin if l.strip()]; best=min(results,key=lambda x:x.get('CER',999)); print(f'Best CER: {best[\"CER\"]:.4f}, WER: {best[\"WER\"]:.4f}')" 2>/dev/null
fi
```

---

## Option 1: Papermill (Recommended)

Runs notebooks directly without conversion.

```bash
pip install papermill
nohup papermill <notebook>.ipynb output.ipynb > papermill.log 2>&1 &
```

Monitor:
```bash
tail -f papermill.log
```

## Option 2: Convert to Python Script

```bash
jupyter nbconvert --to script <notebook>.ipynb
nohup python <notebook>.py > output.log 2>&1 &
```

**Note:** `%pip install` magic commands need manual removal before running as `.py`

## Important Notes

- Ray Tune notebooks require the OCR service running first (Docker)
- For Ray workers, imports must be inside trainable functions

## Example: Ray Tune PaddleOCR

```bash
# 1. Start OCR service
cd src/paddle_ocr && docker compose up -d ocr-cpu

# 2. Run notebook with papermill
cd src
nohup papermill paddle_ocr_raytune_rest.ipynb output_raytune.ipynb > papermill.log 2>&1 &

# 3. Monitor
tail -f papermill.log
```