> ## Documentation Index > Fetch the complete documentation index at: https://mirage-eval.com/llms.txt > Use this file to discover all available pages before exploring further. # Run on an HPC cluster > Prepare Mirage, model weights, and datasets, then submit reproducible GPU evaluations with SLURM. This guide covers a first GPU run and repeatable model comparisons on a SLURM cluster. Cluster module names, partitions, GPU types, storage paths, and resource limits differ, so check your center's instructions and edit `slurm/demo.slurm` for local requirements. ## What you need before submitting * A clone of Mirage on storage visible to the compute nodes. * A Python environment with the pinned packages in `requirements.txt`. * Dataset annotations and images arranged as described in [Datasets](/datasets). * The model weights already available in the Hugging Face cache or on cluster storage. Nodes without internet access cannot download weights during the job. * A GPU allocation sized for the selected checkpoint and evaluation. The sample SLURM request is a starting point, not a guarantee that a model fits. The repository currently exposes `paligemma` and `qwen2vl` to `demo.py`. See [Metrics and models](/metrics-and-models) for checkpoint defaults and overrides. DeepSeek-VL is not currently implemented as a Mirage model backend, so it cannot be selected with `--model` or `MIRAGE_VL_MODEL`. ## Clone and prepare the environment Clone the repository to a project or work directory that compute nodes can read: ```bash theme={null} git clone https://github.com/Advanced-Vision-and-Learning-Lab/mirage.git cd mirage ``` Load the Python and CUDA modules required by your center, then create the environment. Match the PyTorch installation to the cluster's supported CUDA setup. The pinned versions are in `requirements.txt`. ```bash theme={null} module load Python/3.11 module load CUDA/12.4 python3 -m venv .venv source .venv/bin/activate python -m pip install --upgrade pip python -m pip install -r requirements.txt ``` Module names and versions above are examples. If the cluster provides a supported PyTorch environment, follow its installation instructions and install any remaining project requirements into that environment. Check that the environment can see the GPU and that the CLI is available: ```bash theme={null} python -c "import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0) if torch.cuda.is_available() else 'CPU')" python demo.py --help ``` ## Stage data and model weights By default, Mirage expects COCO validation data under `datasets/coco/` or Visual Genome data under `datasets/visual_genome/`. Large files are intentionally excluded from Git. If your files live elsewhere, pass `--annotations` and `--images` to `demo.py`; for the sample SLURM script, edit the command to add those options or place data at the default paths. Set a persistent Hugging Face cache path on storage visible to compute nodes. If outbound downloads are disabled, download or stage the model snapshot on a login or transfer node first, then point the corresponding model path variable to that snapshot. ```bash theme={null} export MIRAGE_HF_HOME=/path/to/shared/huggingface export MIRAGE_QWEN_MODEL_PATH=/path/to/models/Qwen2-VL-2B-Instruct ``` For PaliGemma, use `MIRAGE_PALIGEMMA_MODEL_PATH`. The default is a Hugging Face model identifier, which requires the model files to be cached or reachable. Some gated model repositories also require access approval and Hugging Face authentication before downloading. ## Submit a small Qwen2-VL run The sample job runs five COCO images by default. Set these variables before submitting to request a small Qwen2-VL CHAIR run: ```bash theme={null} export MIRAGE_DATASET=coco export MIRAGE_METRIC=chair export MIRAGE_VL_MODEL=qwen2vl export MIRAGE_SAMPLES=5 export MIRAGE_OUTPUT=results/qwen-coco-chair.json export MIRAGE_CACHE_DIR=/path/to/scratch/mirage-cache/qwen-coco-chair sbatch slurm/demo.slurm ``` For Visual Genome, set `MIRAGE_DATASET=visual_genome`. To run another supported metric, set `MIRAGE_METRIC` to `pope`, `rope`, `throne`, or `faithscore` as well as `chair`. FaithScore has extra requirements; see [Metrics and models](/metrics-and-models). NOPE uses a separate input flow documented in [Running evaluations](/running-evaluations). The job writes standard output and errors to `mirage_.out` and `mirage_.err`, and the evaluation result to the path passed through `MIRAGE_OUTPUT`. Check the generated JSON and job logs after completion. Keep result paths unique across jobs to avoid overwriting earlier runs. ## Compare model configurations Run separate jobs for each supported model and metric combination, changing `MIRAGE_VL_MODEL`, the model path variable, and the output path each time. For example, to compare PaliGemma and Qwen2-VL on the same data and metric: ```bash theme={null} export MIRAGE_DATASET=coco export MIRAGE_METRIC=chair export MIRAGE_SAMPLES=100 export MIRAGE_VL_MODEL=paligemma export MIRAGE_OUTPUT=results/paligemma-coco-chair.json export MIRAGE_CACHE_DIR=/path/to/scratch/mirage-cache/paligemma-coco-chair sbatch slurm/demo.slurm export MIRAGE_VL_MODEL=qwen2vl export MIRAGE_OUTPUT=results/qwen-coco-chair.json export MIRAGE_CACHE_DIR=/path/to/scratch/mirage-cache/qwen-coco-chair sbatch slurm/demo.slurm ``` For an initial comparison, keep the dataset, metric, sample count, and other evaluation options the same. The current built-in model keys are listed in [Metrics and models](/metrics-and-models). Each job can use a distinct `MIRAGE_CACHE_DIR`. Cache keys include model identity, prompt, image, and generation settings, so cached responses are reused only when those inputs match. SQLite caches work best when each concurrent job writes to its own database on storage suitable for SQLite. A shared cache can save repeated inference across sequential runs, but avoid multiple active writers to the same cache file unless your filesystem and workload have been validated for that use. ## Adapt the sample job Before using `slurm/demo.slurm` for a full evaluation, edit its `#SBATCH` resource requests and module setup for your cluster. The example requests one GPU, four CPUs, 32 GB of host memory, and four hours. Resource needs depend on the model, metric, image resolution, and sample count. Start with a small pilot and scale the allocation after inspecting GPU memory and run time. For custom dataset paths, extra model arguments, partitions, account codes, or module commands, edit the script's `python demo.py` command and `#SBATCH` block. Then submit it from the Mirage repository root with `sbatch slurm/demo.slurm`. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.