Run on an HPC cluster

Prepare Mirage, model weights, and datasets, then submit reproducible GPU evaluations with SLURM.

This guide covers a first GPU run and repeatable model comparisons on a SLURM cluster. Cluster module names, partitions, GPU types, storage paths, and resource limits differ, so check your center’s instructions and edit slurm/demo.slurm for local requirements.

What you need before submitting

• A clone of Mirage on storage visible to the compute nodes.
• A Python environment with the pinned packages in requirements.txt.
• Dataset annotations and images arranged as described in Datasets.
• The model weights already available in the Hugging Face cache or on cluster storage. Nodes without internet access cannot download weights during the job.
• A GPU allocation sized for the selected checkpoint and evaluation. The sample SLURM request is a starting point, not a guarantee that a model fits.

The repository currently exposes paligemma and qwen2vl to demo.py. See Metrics and models for checkpoint defaults and overrides. DeepSeek-VL is not currently implemented as a Mirage model backend, so it cannot be selected with --model or MIRAGE_VL_MODEL.

Clone and prepare the environment

Clone the repository to a project or work directory that compute nodes can read:

git clone https://github.com/Advanced-Vision-and-Learning-Lab/mirage.git
cd mirage

Load the Python and CUDA modules required by your center, then create the environment. Match the PyTorch installation to the cluster’s supported CUDA setup. The pinned versions are in requirements.txt.

module load Python/3.11
module load CUDA/12.4
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt

Module names and versions above are examples. If the cluster provides a supported PyTorch environment, follow its installation instructions and install any remaining project requirements into that environment.

Check that the environment can see the GPU and that the CLI is available:

python -c "import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0) if torch.cuda.is_available() else 'CPU')"
python demo.py --help

Stage data and model weights

By default, Mirage expects COCO validation data under datasets/coco/ or Visual Genome data under datasets/visual_genome/. Large files are intentionally excluded from Git. If your files live elsewhere, pass --annotations and --images to demo.py; for the sample SLURM script, edit the command to add those options or place data at the default paths.

Set a persistent Hugging Face cache path on storage visible to compute nodes. If outbound downloads are disabled, download or stage the model snapshot on a login or transfer node first, then point the corresponding model path variable to that snapshot.

export MIRAGE_HF_HOME=/path/to/shared/huggingface
export MIRAGE_QWEN_MODEL_PATH=/path/to/models/Qwen2-VL-2B-Instruct

For PaliGemma, use MIRAGE_PALIGEMMA_MODEL_PATH. The default is a Hugging Face model identifier, which requires the model files to be cached or reachable. Some gated model repositories also require access approval and Hugging Face authentication before downloading.

Submit a small Qwen2-VL run

The sample job runs five COCO images by default. Set these variables before submitting to request a small Qwen2-VL CHAIR run:

export MIRAGE_DATASET=coco
export MIRAGE_METRIC=chair
export MIRAGE_VL_MODEL=qwen2vl
export MIRAGE_SAMPLES=5
export MIRAGE_OUTPUT=results/qwen-coco-chair.json
export MIRAGE_CACHE_DIR=/path/to/scratch/mirage-cache/qwen-coco-chair
sbatch slurm/demo.slurm

For Visual Genome, set MIRAGE_DATASET=visual_genome. To run another supported metric, set MIRAGE_METRIC to pope, rope, throne, or faithscore as well as chair. FaithScore has extra requirements; see Metrics and models. NOPE uses a separate input flow documented in Running evaluations.

The job writes standard output and errors to mirage_<job-id>.out and mirage_<job-id>.err, and the evaluation result to the path passed through MIRAGE_OUTPUT. Check the generated JSON and job logs after completion. Keep result paths unique across jobs to avoid overwriting earlier runs.

Compare model configurations

Run separate jobs for each supported model and metric combination, changing MIRAGE_VL_MODEL, the model path variable, and the output path each time. For example, to compare PaliGemma and Qwen2-VL on the same data and metric:

export MIRAGE_DATASET=coco
export MIRAGE_METRIC=chair
export MIRAGE_SAMPLES=100

export MIRAGE_VL_MODEL=paligemma
export MIRAGE_OUTPUT=results/paligemma-coco-chair.json
export MIRAGE_CACHE_DIR=/path/to/scratch/mirage-cache/paligemma-coco-chair
sbatch slurm/demo.slurm

export MIRAGE_VL_MODEL=qwen2vl
export MIRAGE_OUTPUT=results/qwen-coco-chair.json
export MIRAGE_CACHE_DIR=/path/to/scratch/mirage-cache/qwen-coco-chair
sbatch slurm/demo.slurm

For an initial comparison, keep the dataset, metric, sample count, and other evaluation options the same. The current built-in model keys are listed in Metrics and models.

Each job can use a distinct MIRAGE_CACHE_DIR. Cache keys include model identity, prompt, image, and generation settings, so cached responses are reused only when those inputs match. SQLite caches work best when each concurrent job writes to its own database on storage suitable for SQLite. A shared cache can save repeated inference across sequential runs, but avoid multiple active writers to the same cache file unless your filesystem and workload have been validated for that use.

Adapt the sample job

Before using slurm/demo.slurm for a full evaluation, edit its #SBATCH resource requests and module setup for your cluster. The example requests one GPU, four CPUs, 32 GB of host memory, and four hours. Resource needs depend on the model, metric, image resolution, and sample count. Start with a small pilot and scale the allocation after inspecting GPU memory and run time.

For custom dataset paths, extra model arguments, partitions, account codes, or module commands, edit the script’s python demo.py command and #SBATCH block. Then submit it from the Mirage repository root with sbatch slurm/demo.slurm.