Paddle ocr, easyicr and doctr gpu support. (#4)

2026-01-19 17:35:24 +00:00
parent 8e2b7a5096
commit c7ed7b2b9c
105 changed files with 8170 additions and 1263 deletions
--- a/src/paddle_ocr/README.md
+++ b/src/paddle_ocr/README.md
@@ -0,0 +1,824 @@
+# PaddleOCR Tuning REST API
+
+REST API service for PaddleOCR hyperparameter evaluation. Keeps the model loaded in memory for fast repeated evaluations during hyperparameter search.
+
+## Quick Start with Docker Compose
+
+Docker Compose manages building and running containers. The `docker-compose.yml` defines two services:
+- `ocr-cpu` - CPU-only version (works everywhere)
+- `ocr-gpu` - GPU version (requires NVIDIA GPU + Container Toolkit)
+
+### Run CPU Version
+
+```bash
+cd src/paddle_ocr
+
+# Build and start (first time takes ~2-3 min to build, ~30s to load model)
+docker compose up ocr-cpu
+
+# Or run in background (detached)
+docker compose up -d ocr-cpu
+
+# View logs
+docker compose logs -f ocr-cpu
+
+# Stop
+docker compose down
+```
+
+### Run GPU Version
+
+```bash
+# Requires: NVIDIA GPU + nvidia-container-toolkit installed
+docker compose up ocr-gpu
+```
+
+### Test the API
+
+Once running, test with:
+```bash
+# Check health
+curl http://localhost:8000/health
+
+# Or use the test script
+pip install requests
+python test.py --url http://localhost:8000
+```
+
+### What Docker Compose Does
+
+```
+docker compose up ocr-cpu
+       │
+       ├─► Builds image from Dockerfile.cpu (if not exists)
+       ├─► Creates container "paddle-ocr-cpu"
+       ├─► Mounts ../dataset → /app/dataset (your PDF images)
+       ├─► Mounts paddlex-cache volume (persists downloaded models)
+       ├─► Exposes port 8000
+       └─► Runs: uvicorn paddle_ocr_tuning_rest:app --host 0.0.0.0 --port 8000
+```
+
+## Files
+
+| File | Description |
+|------|-------------|
+| `paddle_ocr_tuning_rest.py` | FastAPI REST service |
+| `dataset_manager.py` | Dataset loader |
+| `test.py` | API test client |
+| `Dockerfile.cpu` | CPU-only image (x86_64 + ARM64 with local wheel) |
+| `Dockerfile.gpu` | GPU/CUDA image (x86_64 + ARM64 with local wheel) |
+| `Dockerfile.build-paddle` | PaddlePaddle GPU wheel builder for ARM64 |
+| `Dockerfile.build-paddle-cpu` | PaddlePaddle CPU wheel builder for ARM64 |
+| `docker-compose.yml` | Service orchestration |
+| `docker-compose.cpu-registry.yml` | Pull CPU image from registry |
+| `docker-compose.gpu-registry.yml` | Pull GPU image from registry |
+| `wheels/` | Local PaddlePaddle wheels (created by build-paddle) |
+
+## API Endpoints
+
+### `GET /health`
+Check if service is ready.
+
+```json
+{"status": "ok", "model_loaded": true, "dataset_loaded": true, "dataset_size": 24}
+```
+
+### `POST /evaluate`
+Run OCR evaluation with given hyperparameters.
+
+**Request:**
+```json
+{
+  "pdf_folder": "/app/dataset",
+  "textline_orientation": true,
+  "use_doc_orientation_classify": false,
+  "use_doc_unwarping": false,
+  "text_det_thresh": 0.469,
+  "text_det_box_thresh": 0.5412,
+  "text_det_unclip_ratio": 0.0,
+  "text_rec_score_thresh": 0.635,
+  "start_page": 5,
+  "end_page": 10
+}
+```
+
+**Response:**
+```json
+{"CER": 0.0115, "WER": 0.0989, "TIME": 330.5, "PAGES": 5, "TIME_PER_PAGE": 66.1}
+```
+
+### `POST /evaluate_full`
+Same as `/evaluate` but runs on ALL pages (ignores start_page/end_page).
+
+## Debug Output (debugset)
+
+The `debugset` folder allows saving OCR predictions for debugging and analysis. When `save_output=True` is passed to `/evaluate`, predictions are written to `/app/debugset`.
+
+### Enable Debug Output
+
+```json
+{
+  "pdf_folder": "/app/dataset",
+  "save_output": true,
+  "start_page": 5,
+  "end_page": 10
+}
+```
+
+### Output Structure
+
+```
+debugset/
+├── doc1/
+│   └── paddle_ocr/
+│       ├── page_0005.txt
+│       ├── page_0006.txt
+│       └── ...
+├── doc2/
+│   └── paddle_ocr/
+│       └── ...
+```
+
+Each `.txt` file contains the OCR-extracted text for that page.
+
+### Docker Mount
+
+The `debugset` folder is mounted read-write in docker-compose:
+
+```yaml
+volumes:
+  - ../debugset:/app/debugset:rw
+```
+
+### Use Cases
+
+- **Compare OCR engines**: Run same pages through PaddleOCR, DocTR, EasyOCR with `save_output=True`, then diff results
+- **Debug hyperparameters**: See how different settings affect text extraction
+- **Ground truth comparison**: Compare predictions against expected output
+
+## Building Images
+
+### CPU Image (Multi-Architecture)
+
+```bash
+# Local build (current architecture)
+docker build -f Dockerfile.cpu -t paddle-ocr-api:cpu .
+
+# Multi-arch build with buildx (amd64 + arm64)
+docker buildx create --name multiarch --use
+docker buildx build -f Dockerfile.cpu \
+  --platform linux/amd64,linux/arm64 \
+  -t paddle-ocr-api:cpu \
+  --push .
+```
+
+### GPU Image (x86_64 + ARM64 with local wheel)
+
+```bash
+docker build -f Dockerfile.gpu -t paddle-ocr-api:gpu .
+```
+
+> **Note:** PaddlePaddle GPU 3.x packages are **not on PyPI**. The Dockerfile installs from PaddlePaddle's official CUDA index (`paddlepaddle.org.cn/packages/stable/cu126/`). This is handled automatically during build.
+
+## Running
+
+### CPU (Any machine)
+
+```bash
+docker run -d -p 8000:8000 \
+  -v $(pwd)/../dataset:/app/dataset:ro \
+  -v paddlex-cache:/root/.paddlex \
+  paddle-ocr-api:cpu
+```
+
+### GPU (NVIDIA)
+
+```bash
+docker run -d -p 8000:8000 --gpus all \
+  -v $(pwd)/../dataset:/app/dataset:ro \
+  -v paddlex-cache:/root/.paddlex \
+  paddle-ocr-api:gpu
+```
+
+## GPU Support Analysis
+
+### Host System Reference (DGX Spark)
+
+This section documents GPU support findings based on testing on an NVIDIA DGX Spark:
+
+| Component | Value |
+|-----------|-------|
+| Architecture | ARM64 (aarch64) |
+| CPU | NVIDIA Grace (ARM) |
+| GPU | NVIDIA GB10 |
+| CUDA Version | 13.0 |
+| Driver | 580.95.05 |
+| OS | Ubuntu 24.04 LTS |
+| Container Toolkit | nvidia-container-toolkit 1.18.1 |
+| Docker | 28.5.1 |
+| Docker Compose | v2.40.0 |
+
+### PaddlePaddle GPU Platform Support
+
+**Note:** PaddlePaddle-GPU does NOT have prebuilt ARM64 wheels on PyPI, but ARM64 support is available via custom-built wheels.
+
+| Platform | CPU | GPU |
+|----------|-----|-----|
+| Linux x86_64 | ✅ | ✅ CUDA 10.2/11.x/12.x |
+| Windows x64 | ✅ | ✅ CUDA 10.2/11.x/12.x |
+| macOS x64 | ✅ | ❌ |
+| macOS ARM64 (M1/M2) | ✅ | ❌ |
+| Linux ARM64 (Jetson/DGX) | ✅ | ⚠️ Limited - see Blackwell note |
+
+**Source:** [PaddlePaddle-GPU PyPI](https://pypi.org/project/paddlepaddle-gpu/) - only `manylinux_x86_64` and `win_amd64` wheels available on PyPI. ARM64 wheels must be built from source or downloaded from Gitea packages.
+
+### ARM64 GPU Support
+
+ARM64 GPU support is available but requires custom-built wheels:
+
+1. **No prebuilt PyPI wheels**: `pip install paddlepaddle-gpu` fails on ARM64 - no compatible wheels exist on PyPI
+2. **Custom wheels work**: This project provides Dockerfiles to build ARM64 GPU wheels from source
+3. **CI/CD builds ARM64 GPU images**: Pre-built wheels are available from Gitea packages
+
+**To use GPU on ARM64:**
+- Use the pre-built images from the container registry, or
+- Build the wheel locally using `Dockerfile.build-paddle` (see Option 2 below), or
+- Download the wheel from Gitea packages: `wheels/paddlepaddle_gpu-3.0.0-cp311-cp311-linux_aarch64.whl`
+
+### ⚠️ Known Limitation: Blackwell GPU (sm_121 / GB10)
+
+**Status: GPU inference does NOT work on NVIDIA Blackwell GPUs (DGX Spark, GB200, etc.)**
+
+#### Symptoms
+
+When running PaddleOCR on Blackwell GPUs:
+- CUDA loads successfully ✅
+- Basic tensor operations work ✅
+- **Detection model outputs constant values** ❌
+- 0 text regions detected
+- CER/WER = 100% (nothing recognized)
+
+#### Root Cause
+
+**Confirmed:** PaddlePaddle's entire CUDA backend does NOT support Blackwell (sm_121). This is NOT just an inference model problem - even basic operations fail.
+
+**Test Results (January 2026):**
+
+1. **PTX JIT Test** (`CUDA_FORCE_PTX_JIT=1`):
+   ```
+   OSError: CUDA error(209), no kernel image is available for execution on the device.
+   [Hint: 'cudaErrorNoKernelImageForDevice']
+   ```
+   → Confirmed: No PTX code exists in PaddlePaddle binaries
+
+2. **Dynamic Graph Mode Test** (bypassing inference models):
+   ```
+   Conv2D + BatchNorm output:
+     Output min: 0.0000
+     Output max: 0.0000
+     Output mean: 0.0000
+   Dynamic graph mode: BROKEN (constant output)
+   ```
+   → Confirmed: Even simple nn.Conv2D produces zeros on Blackwell
+
+**Conclusion:** The issue is PaddlePaddle's compiled CUDA kernels (cubins), not just the inference models. The entire framework was compiled without sm_121 support and without PTX for JIT compilation.
+
+**Why building PaddlePaddle from source doesn't fix it:**
+
+1. ⚠️ Building with `CUDA_ARCH=121` requires CUDA 13.0+ (PaddlePaddle only supports up to CUDA 12.6)
+2. ❌ Even if you could build it, PaddleOCR models contain pre-compiled CUDA ops
+3. ❌ These model files were exported/compiled targeting sm_80/sm_90 architectures
+4. ❌ The model kernels execute on GPU but produce garbage output on sm_121
+
+**To truly fix this**, the PaddlePaddle team would need to:
+1. Add sm_121 to their model export pipeline
+2. Re-export all PaddleOCR models (PP-OCRv4, PP-OCRv5, etc.) with Blackwell support
+3. Release new model versions
+
+This is tracked in [GitHub Issue #17327](https://github.com/PaddlePaddle/PaddleOCR/issues/17327).
+
+#### Debug Script
+
+Use the included debug script to verify this issue:
+
+```bash
+docker exec paddle-ocr-gpu python /app/scripts/debug_gpu_detection.py /app/dataset/0/img/page_0001.png
+```
+
+Expected output showing the problem:
+```
+OUTPUT ANALYSIS:
+  Shape: (1, 1, 640, 640)
+  Min: 0.000010
+  Max: 0.000010   # <-- Same as min = constant output
+  Mean: 0.000010
+
+DIAGNOSIS:
+  PROBLEM: Output is constant - model inference is broken!
+  This typically indicates GPU compute capability mismatch.
+```
+
+#### Workarounds
+
+1. **Use CPU mode** (recommended):
+   ```bash
+   docker compose up ocr-cpu
+   ```
+   The ARM Grace CPU is fast (~2-5 sec/page). This is the reliable option.
+
+2. **Use EasyOCR or DocTR with GPU**:
+   These use PyTorch which has official ARM64 CUDA wheels (cu128 index):
+   ```bash
+   # EasyOCR with GPU on DGX Spark
+   docker build -f ../easyocr_service/Dockerfile.gpu -t easyocr-gpu ../easyocr_service
+   docker run --gpus all -p 8002:8000 easyocr-gpu
+   ```
+
+3. **Wait for PaddlePaddle Blackwell support**:
+   Track [GitHub Issue #17327](https://github.com/PaddlePaddle/PaddleOCR/issues/17327) for updates.
+
+#### GPU Support Matrix (Updated)
+
+| GPU Architecture | Compute | CPU | GPU |
+|------------------|---------|-----|-----|
+| Ampere (A100, A10) | sm_80 | ✅ | ✅ |
+| Hopper (H100, H200) | sm_90 | ✅ | ✅ |
+| **Blackwell (GB10, GB200)** | sm_121 | ✅ | ❌ Not supported |
+
+#### FAQ: Why Doesn't CUDA Backward Compatibility Work?
+
+**Q: CUDA normally runs older kernels on newer GPUs. Why doesn't this work for Blackwell?**
+
+Per [NVIDIA Blackwell Compatibility Guide](https://docs.nvidia.com/cuda/blackwell-compatibility-guide/):
+
+CUDA **can** run older code on newer GPUs via **PTX JIT compilation**:
+1. PTX (Parallel Thread Execution) is NVIDIA's intermediate representation
+2. If an app includes PTX code, the driver JIT-compiles it for the target GPU
+3. This allows sm_80 code to run on sm_121
+
+**The problem**: PaddleOCR inference models contain only pre-compiled **cubins** (SASS binary), not PTX. Without PTX, there's nothing to JIT-compile.
+
+We tested PTX JIT (January 2026):
+```bash
+# Force PTX JIT compilation
+docker run --gpus all -e CUDA_FORCE_PTX_JIT=1 paddle-ocr-gpu \
+  python /app/scripts/debug_gpu_detection.py /app/dataset/0/img/page_0001.png
+
+# Result:
+# OSError: CUDA error(209), no kernel image is available for execution on the device.
+```
+**Confirmed: No PTX exists** in PaddlePaddle binaries. The CUDA kernels are cubins-only (SASS binary), compiled for sm_80/sm_90 without PTX fallback.
+
+**Note on sm_121**: Per NVIDIA docs, "sm_121 is the same as sm_120 since the only difference is physically integrated CPU+GPU memory of Spark." The issue is general Blackwell (sm_12x) support, not Spark-specific.
+
+#### FAQ: Does Dynamic Graph Mode Work on Blackwell?
+
+**Q: Can I bypass inference models and use PaddlePaddle's dynamic graph mode?**
+
+**No.** We tested dynamic graph mode (January 2026):
+```bash
+# Test script runs: paddle.nn.Conv2D + paddle.nn.BatchNorm2D
+python /app/scripts/test_dynamic_mode.py
+
+# Result:
+# Input shape: [1, 3, 224, 224]
+# Output shape: [1, 64, 112, 112]
+# Output min: 0.0000
+# Output max: 0.0000  # <-- All zeros!
+# Output mean: 0.0000
+# Dynamic graph mode: BROKEN (constant output)
+```
+
+**Conclusion:** The problem isn't limited to inference models. PaddlePaddle's core CUDA kernels (Conv2D, BatchNorm, etc.) produce garbage on sm_121. The entire framework lacks Blackwell support.
+
+#### FAQ: Can I Run AMD64 Containers on ARM64 DGX Spark?
+
+**Q: Can I just run the working x86_64 GPU image via emulation?**
+
+**Short answer: Yes for CPU, No for GPU.**
+
+You can run amd64 containers via QEMU emulation:
+```bash
+# Install QEMU
+sudo apt-get install qemu binfmt-support qemu-user-static
+docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
+
+# Run amd64 container
+docker run --platform linux/amd64 paddle-ocr-gpu:amd64 ...
+```
+
+**But GPU doesn't work:**
+- QEMU emulates CPU instructions (x86 → ARM)
+- **QEMU user-mode does NOT support GPU passthrough**
+- GPU calls from emulated x86 code cannot reach the ARM64 GPU
+
+So even if the amd64 image works on x86_64:
+- ❌ No GPU access through QEMU
+- ❌ CPU emulation is 10-100x slower than native ARM64
+- ❌ Defeats the purpose entirely
+
+| Approach | CPU | GPU | Speed |
+|----------|-----|-----|-------|
+| ARM64 native (CPU) | ✅ | N/A | Fast (~2-5s/page) |
+| ARM64 native (GPU) | ✅ | ❌ Blackwell issue | - |
+| AMD64 via QEMU | ⚠️ Works | ❌ No passthrough | 10-100x slower |
+
+### Options for ARM64 Systems
+
+#### Option 1: CPU-Only (Recommended)
+
+Use `Dockerfile.cpu` which works on ARM64:
+
+```bash
+# On DGX Spark
+docker compose up ocr-cpu
+
+# Or build directly
+docker build -f Dockerfile.cpu -t paddle-ocr-api:cpu .
+```
+
+**Performance:** CPU inference on ARM64 Grace is surprisingly fast due to high core count. Expect ~2-5 seconds per page.
+
+#### Option 2: Build PaddlePaddle from Source (Docker-based)
+
+Use the included Docker builder to compile PaddlePaddle GPU for ARM64:
+
+```bash
+cd src/paddle_ocr
+
+# Step 1: Build the PaddlePaddle GPU wheel (one-time, 2-4 hours)
+docker compose --profile build run --rm build-paddle
+
+# Verify wheel was created
+ls -la wheels/paddlepaddle*.whl
+
+# Step 2: Build the GPU image (uses local wheel)
+docker compose build ocr-gpu
+
+# Step 3: Run with GPU
+docker compose up ocr-gpu
+
+# Verify GPU is working
+docker compose exec ocr-gpu python -c "import paddle; print(paddle.device.is_compiled_with_cuda())"
+```
+
+**What this does:**
+1. `build-paddle` compiles PaddlePaddle from source inside a CUDA container
+2. The wheel is saved to `./wheels/` directory
+3. `Dockerfile.gpu` detects the local wheel and uses it instead of PyPI
+
+**Caveats:**
+- Build takes 2-4 hours on first run
+- Requires ~20GB disk space during build
+- Not officially supported by PaddlePaddle team
+- May need adjustments for future PaddlePaddle versions
+
+See: [GitHub Issue #17327](https://github.com/PaddlePaddle/PaddleOCR/issues/17327)
+
+#### Option 3: Alternative OCR Engines
+
+For ARM64 GPU acceleration, consider alternatives:
+
+| Engine | ARM64 GPU | Notes |
+|--------|-----------|-------|
+| **Tesseract** | ❌ CPU-only | Good fallback, widely available |
+| **EasyOCR** | ⚠️ Via PyTorch | PyTorch has ARM64 GPU support |
+| **TrOCR** | ⚠️ Via Transformers | Hugging Face Transformers + PyTorch |
+| **docTR** | ⚠️ Via TensorFlow/PyTorch | Both backends have ARM64 support |
+
+EasyOCR with PyTorch is a viable alternative:
+```bash
+pip install torch torchvision --index-url https://download.pytorch.org/whl/cu121
+pip install easyocr
+```
+
+### x86_64 GPU Setup (Working)
+
+For x86_64 systems with NVIDIA GPU, the GPU Docker works:
+
+```bash
+# Verify GPU is accessible
+nvidia-smi
+
+# Verify Docker GPU access
+docker run --rm --gpus all nvidia/cuda:12.0-base nvidia-smi
+
+# Build and run GPU version
+docker compose up ocr-gpu
+```
+
+### GPU Docker Compose Configuration
+
+The `docker-compose.yml` configures GPU access via:
+
+```yaml
+deploy:
+  resources:
+    reservations:
+      devices:
+        - driver: nvidia
+          count: 1
+          capabilities: [gpu]
+```
+
+This requires Docker Compose v2 and nvidia-container-toolkit.
+
+## DGX Spark / ARM64 Quick Start
+
+For ARM64 systems (DGX Spark, Jetson, Graviton), use CPU-only:
+
+```bash
+cd src/paddle_ocr
+
+# Build ARM64-native CPU image
+docker build -f Dockerfile.cpu -t paddle-ocr-api:arm64 .
+
+# Run
+docker run -d -p 8000:8000 \
+  -v $(pwd)/../dataset:/app/dataset:ro \
+  paddle-ocr-api:arm64
+
+# Test
+curl http://localhost:8000/health
+```
+
+### Cross-Compile from x86_64
+
+Build ARM64 images from an x86_64 machine:
+
+```bash
+# Setup buildx for multi-arch
+docker buildx create --name mybuilder --use
+
+# Build ARM64 image from x86_64 machine
+docker buildx build -f Dockerfile.cpu \
+  --platform linux/arm64 \
+  -t paddle-ocr-api:arm64 \
+  --load .
+
+# Save and transfer to DGX Spark
+docker save paddle-ocr-api:arm64 | gzip > paddle-ocr-arm64.tar.gz
+scp paddle-ocr-arm64.tar.gz dgx-spark:~/
+
+# On DGX Spark:
+docker load < paddle-ocr-arm64.tar.gz
+```
+
+## Using with Ray Tune
+
+### Multi-Worker Setup for Parallel Trials
+
+Run multiple workers for parallel hyperparameter tuning:
+
+```bash
+cd src/paddle_ocr
+
+# Start 2 CPU workers (ports 8001-8002)
+sudo docker compose -f docker-compose.workers.yml --profile cpu up -d
+
+# Or for GPU workers (if supported)
+sudo docker compose -f docker-compose.workers.yml --profile gpu up -d
+
+# Check workers are healthy
+curl http://localhost:8001/health
+curl http://localhost:8002/health
+```
+
+Then run the notebook with `max_concurrent_trials=2` to use both workers in parallel.
+
+### Single Worker Setup
+
+Update your notebook's `trainable_paddle_ocr` function:
+
+```python
+import requests
+
+API_URL = "http://localhost:8000/evaluate"
+
+def trainable_paddle_ocr(config):
+    """Call OCR API instead of subprocess."""
+    payload = {
+        "pdf_folder": "/app/dataset",
+        "use_doc_orientation_classify": config.get("use_doc_orientation_classify", False),
+        "use_doc_unwarping": config.get("use_doc_unwarping", False),
+        "textline_orientation": config.get("textline_orientation", True),
+        "text_det_thresh": config.get("text_det_thresh", 0.0),
+        "text_det_box_thresh": config.get("text_det_box_thresh", 0.0),
+        "text_det_unclip_ratio": config.get("text_det_unclip_ratio", 1.5),
+        "text_rec_score_thresh": config.get("text_rec_score_thresh", 0.0),
+    }
+
+    try:
+        response = requests.post(API_URL, json=payload, timeout=600)
+        response.raise_for_status()
+        metrics = response.json()
+        tune.report(metrics=metrics)
+    except Exception as e:
+        tune.report({"CER": 1.0, "WER": 1.0, "ERROR": str(e)[:500]})
+```
+
+## Architecture: Model Lifecycle
+
+The model is loaded **once** at container startup and stays in memory for all requests:
+
+```mermaid
+flowchart TB
+    subgraph Container["Docker Container Lifecycle"]
+        Start([Container Start]) --> Load[Load PaddleOCR Models<br/>~10-30s one-time cost]
+        Load --> Ready[API Ready<br/>Models in RAM ~500MB]
+
+        subgraph Requests["Incoming Requests - Models Stay Loaded"]
+            Ready --> R1[Request 1] --> Ready
+            Ready --> R2[Request 2] --> Ready
+            Ready --> RN[Request N...] --> Ready
+        end
+
+        Ready --> Stop([Container Stop])
+        Stop --> Free[Models Freed]
+    end
+
+    style Load fill:#f9f,stroke:#333
+    style Ready fill:#9f9,stroke:#333
+    style Requests fill:#e8f4ea,stroke:#090
+```
+
+**Subprocess vs REST API comparison:**
+
+```mermaid
+flowchart LR
+    subgraph Subprocess["❌ Subprocess Approach"]
+        direction TB
+        S1[Trial 1] --> L1[Load Model ~10s]
+        L1 --> E1[Evaluate ~60s]
+        E1 --> U1[Unload]
+        U1 --> S2[Trial 2]
+        S2 --> L2[Load Model ~10s]
+        L2 --> E2[Evaluate ~60s]
+    end
+
+    subgraph REST["✅ REST API Approach"]
+        direction TB
+        Start2[Start Container] --> Load2[Load Model ~10s]
+        Load2 --> Ready2[Model in Memory]
+        Ready2 --> T1[Trial 1 ~60s]
+        T1 --> Ready2
+        Ready2 --> T2[Trial 2 ~60s]
+        T2 --> Ready2
+        Ready2 --> TN[Trial N ~60s]
+    end
+
+    style L1 fill:#faa
+    style L2 fill:#faa
+    style Load2 fill:#afa
+    style Ready2 fill:#afa
+```
+
+## Performance Comparison
+
+| Approach | Model Load | Per-Trial Overhead | 64 Trials |
+|----------|------------|-------------------|-----------|
+| Subprocess (original) | Every trial (~10s) | ~10s | ~7 hours |
+| Docker per trial | Every trial (~10s) | ~12-15s | ~7.5 hours |
+| **REST API** | **Once** | **~0.1s** | **~5.8 hours** |
+
+The REST API saves ~1+ hour by loading the model only once.
+
+## Troubleshooting
+
+### Model download slow on first run
+The first run downloads ~500MB of models. Use volume `paddlex-cache` to persist them.
+
+### Out of memory
+Reduce `max_concurrent_trials` in Ray Tune, or increase container memory:
+```bash
+docker run --memory=8g ...
+```
+
+### GPU not detected
+Ensure NVIDIA Container Toolkit is installed:
+```bash
+nvidia-smi  # Should work
+docker run --rm --gpus all nvidia/cuda:12.0-base nvidia-smi  # Should work
+```
+
+### PaddlePaddle GPU installation fails
+PaddlePaddle 3.x GPU packages are **not available on PyPI**. They must be installed from PaddlePaddle's official index:
+```bash
+# For CUDA 12.x
+pip install paddlepaddle-gpu==3.2.0 -i https://www.paddlepaddle.org.cn/packages/stable/cu126/
+
+# For CUDA 11.8
+pip install paddlepaddle-gpu==3.2.0 -i https://www.paddlepaddle.org.cn/packages/stable/cu118/
+```
+The Dockerfile.gpu handles this automatically.
+
+## CI/CD Pipeline
+
+The project includes a Gitea Actions workflow (`.gitea/workflows/ci.yaml`) for automated builds.
+
+### What CI Builds
+
+| Image | Architecture | Source |
+|-------|--------------|--------|
+| `paddle-ocr-cpu:amd64` | amd64 | PyPI paddlepaddle |
+| `paddle-ocr-cpu:arm64` | arm64 | Pre-built wheel from Gitea packages |
+| `paddle-ocr-gpu:amd64` | amd64 | PyPI paddlepaddle-gpu |
+| `paddle-ocr-gpu:arm64` | arm64 | Pre-built wheel from Gitea packages |
+
+### ARM64 Wheel Workflow
+
+Since PyPI wheels don't work on ARM64 (x86 SSE instructions), wheels must be built from source using sse2neon:
+
+1. Built manually on an ARM64 machine (one-time)
+2. Uploaded to Gitea generic packages
+3. Downloaded by CI when building ARM64 images
+
+#### Step 1: Build ARM64 Wheels (One-time, on ARM64 machine)
+
+```bash
+cd src/paddle_ocr
+
+# Build GPU wheel (requires NVIDIA GPU, takes 1-2 hours)
+sudo docker build -t paddle-builder:gpu-arm64 -f Dockerfile.build-paddle .
+sudo docker run --rm -v ./wheels:/wheels paddle-builder:gpu-arm64
+
+# Build CPU wheel (no GPU required, takes 1-2 hours)
+sudo docker build -t paddle-builder:cpu-arm64 -f Dockerfile.build-paddle-cpu .
+sudo docker run --rm -v ./wheels:/wheels paddle-builder:cpu-arm64
+
+# Verify wheels were created
+ls -la wheels/paddlepaddle*.whl
+# paddlepaddle_gpu-3.0.0-cp311-cp311-linux_aarch64.whl (GPU)
+# paddlepaddle-3.0.0-cp311-cp311-linux_aarch64.whl (CPU)
+```
+
+#### Step 2: Upload Wheels to Gitea Packages
+
+```bash
+export GITEA_TOKEN="your-token-here"
+
+# Upload GPU wheel
+curl -X PUT \
+  -H "Authorization: token $GITEA_TOKEN" \
+  --upload-file wheels/paddlepaddle_gpu-3.0.0-cp311-cp311-linux_aarch64.whl \
+  "https://seryus.ddns.net/api/packages/unir/generic/paddlepaddle-gpu-arm64/3.0.0/paddlepaddle_gpu-3.0.0-cp311-cp311-linux_aarch64.whl"
+
+# Upload CPU wheel
+curl -X PUT \
+  -H "Authorization: token $GITEA_TOKEN" \
+  --upload-file wheels/paddlepaddle-3.0.0-cp311-cp311-linux_aarch64.whl \
+  "https://seryus.ddns.net/api/packages/unir/generic/paddlepaddle-cpu-arm64/3.0.0/paddlepaddle-3.0.0-cp311-cp311-linux_aarch64.whl"
+```
+
+Wheels available at:
+```
+https://seryus.ddns.net/api/packages/unir/generic/paddlepaddle-gpu-arm64/3.0.0/paddlepaddle_gpu-3.0.0-cp311-cp311-linux_aarch64.whl
+https://seryus.ddns.net/api/packages/unir/generic/paddlepaddle-cpu-arm64/3.0.0/paddlepaddle-3.0.0-cp311-cp311-linux_aarch64.whl
+```
+
+#### Step 3: CI Builds Images
+
+CI automatically:
+1. Downloads ARM64 wheels from Gitea packages (for arm64 builds only)
+2. Builds both CPU and GPU images for amd64 and arm64
+3. Pushes to registry with arch-specific tags
+
+### Required CI Secrets
+
+Configure these in Gitea repository settings:
+
+| Secret | Description |
+|--------|-------------|
+| `CI_READWRITE` | Gitea token with registry read/write access |
+
+### Manual Image Push
+
+```bash
+# Login to registry
+docker login seryus.ddns.net
+
+# Build and push CPU (multi-arch)
+docker buildx build -f Dockerfile.cpu \
+  --platform linux/amd64,linux/arm64 \
+  -t seryus.ddns.net/unir/paddle-ocr-api:cpu \
+  --push .
+
+# Build and push GPU (x86_64)
+docker build -f Dockerfile.gpu -t seryus.ddns.net/unir/paddle-ocr-api:gpu-amd64 .
+docker push seryus.ddns.net/unir/paddle-ocr-api:gpu-amd64
+
+# Build and push GPU (ARM64) - requires wheel in wheels/
+docker buildx build -f Dockerfile.gpu \
+  --platform linux/arm64 \
+  -t seryus.ddns.net/unir/paddle-ocr-api:gpu-arm64 \
+  --push .
+```
+
+### Updating the ARM64 Wheels
+
+When PaddlePaddle releases a new version:
+
+1. Update `PADDLE_VERSION` in `Dockerfile.build-paddle` and `Dockerfile.build-paddle-cpu`
+2. Rebuild both wheels on an ARM64 machine
+3. Upload to Gitea packages with new version
+4. Update `PADDLE_VERSION` in `.gitea/workflows/ci.yaml`