---
license: mit
base_model:
- microsoft/Phi-4-multimodal-instruct
pipeline_tag: image-text-to-text
tags:
- biomedical
- multimodal
- chest-x-ray
- vlm
- phi-4
- supervised-finetuning
---

# CheXPO-v2: Preference Optimization for Chest X-ray VLMs with Knowledge Graph Consistency

<div align="center">
<a href="https://github.com/ecoxial2007/CheX-Phi4MM">【GitHub Project】</a> |
<a href="https://arxiv.org/abs/2512.17213">【Arxiv Paper】</a>
</div>

## Introduction

**CheXPO-v2** is a state-of-the-art multimodal vision-language model specialized for chest X-ray interpretation. It is built upon the Phi-4 multimodal architecture and refined using a novel alignment framework that shifts from outcome-based rewards to process supervision.

By integrating a **Knowledge Graph (KG) Consistency Reward**, CheXPO-v2 ensures that the model's reasoning process is clinically sound, significantly reducing hallucinations and improving factual accuracy in radiology report generation and visual question answering.

## Key Innovations

- **Process Supervision via KG Consistency**: Unlike traditional RLHF (e.g., GRPO) that only rewards the final answer, CheXPO-v2 uses **Entity-Relation Matching** to penalize incoherent logic at the atomic level (Disease, Relation, Anatomy).
- **Data Efficiency**: Achieves new SOTA performance on benchmarks like MIMIC-CXR-VQA using only **5k preference samples**.
- **Verifiable Reasoning**: Produces structured and clinically verifiable reasoning chains, mitigating the "overthinking" problem where models generate verbose but flawed logic.

## How to Use

This model is compatible with the `transformers` library (version >= 4.48.2). You can use the standard `AutoModelForCausalLM` and `AutoProcessor` to run inference.

```python
import torch
from transformers import AutoModelForCausalLM, AutoProcessor
from PIL import Image

# Load model and processor
model_id = "ecoxial2007/CheX-Phi4MM-SFT"
model = AutoModelForCausalLM.from_pretrained(
model_id,
device_map="auto",
trust_remote_code=True,
torch_dtype=torch.bfloat16
)
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)

# Prepare input
prompt = "<|user|>\n<|image_1|>\nWhat are the main findings in this chest X-ray? <|end|>\n<|assistant|>\n"
image = Image.open("path_to_your_cxr_image.jpg")

inputs = processor(text=prompt, images=[image], return_tensors="pt").to(model.device)

# Generate response
outputs = model.generate(**inputs, max_new_tokens=512)
response = processor.batch_decode(outputs, skip_special_tokens=True)
print(response[0])

```

## Citation

If you find this work useful, please cite our paper:

```


@article
{liang2024chexpo2,
title={CheXPO-v2: Preference Optimization for Chest X-ray VLMs with Knowledge Graph Consistency},
author={Liang, Xiao and An, Yuxuan and Wang, Di and Hu, Jiawei and Jiao, Zhicheng and Jing, Bin and Wang, Quan},
journal={arXiv preprint arXiv:2512.17213},
year={2024}
}
```

## Acknowledgement

This project is hosted at [ecoxial2007/CheX-Phi4MM](https://github.com/ecoxial2007/CheX-Phi4MM). We thank the community for their contributions to medical multimodal AI.