Metrics and models

Implemented metric paths and model backends available in the demo.

Metrics

The demo exposes these metric choices:

Metric	What the implementation evaluates	Notes
CHAIR	Object mentions in generated image captions against annotated objects.	Uses COCO synonym handling for COCO; Visual Genome uses its object names.
POPE	Yes/no answers to questions about objects present or absent in an image.	The demo uses random negative-object sampling by default.
ROPE	Object recognition for numbered, boxed image regions.	The demo uses five boxes and asks the model to identify each region.
THRONE	Caption object predictions against the dataset’s object categories.	Uses the loaded dataset’s category or object-name list.
FaithScore	Faithfulness checks on generated captions.	Requires a local LLaMA model path via --llama-model or MIRAGE_LLAMA_MODEL_PATH, plus a PaliGemma verification model.
NOPE	Scores responses to the NOPE question set.	Requires pre-generated Turn 2 responses via --nope-turn2; Turn 1 input is optional.

Metric availability can depend on the dataset path and local files. For example, ROPE needs annotated boxes, while NOPE reads its response JSON input rather than the normal dataset caption path.

Models

The prediction factory supports two local vision-language model wrappers:

• paligemma uses google/paligemma-3b-mix-224 by default. Override the model identifier or local path with MIRAGE_PALIGEMMA_MODEL_PATH.
• qwen2vl uses Qwen/Qwen2-VL-2B-Instruct by default. Override it with MIRAGE_QWEN_MODEL_PATH.

These are the model keys currently supported by demo.py. A different checkpoint can be used through the model-path override only when it is compatible with that wrapper. DeepSeek-VL is not currently supported; adding it requires a model wrapper and factory registration before it can be selected as a separate model key. See Maintain the docs for the implementation handoff requirements.

Set MIRAGE_VL_MODEL to choose the default model when --model is omitted. Models are loaded for inference at runtime; model weights are not included in the repository.

Result caching

Predictions pass through the predict package, which wraps model inference with a response cache. Cache identity and storage are implemented in predict/identity.py and predict/cache.py; the demo’s SLURM example sets MIRAGE_CACHE_DIR to a job-specific directory.