Abstract

In the context of Vision-Language Models (VLMs), adversarial examples (or attacks) are image and prompt pairs which, when fed to a target model, make it hallucinate, disclose information it shouldn’t, or misclassify information. In this framework, there has been a growing interest in entropy-based attacks (see e.g., [1, 2]); that is, attack strategies which aim to force the model to change its decision on high-entropy tokens in the output, instead of pushing a change in the whole answer.

A recent paper [2] proved that changing less than 20% of the tokens is enough to measure an increase in the Attack Success Rate (ASR). Current VLM architectures are built to take patchified images and text as input. Both the patches and the text tokens are encoded together in a joint embedding space; finally, the encoded information is used to produce the
output.

The core idea of the project is to investigate entropy-based attacks and search for relations with existing a priori entropy in image patches. We aim to determine if it is possible to relate patch entropy with generation entropy, and hence build attacks that aim to influence high entropy tokens by acting on single patches instead of the whole image (using, for example, PGD, image transformations, or patch substitutions). This project will investigate the vulnerability of state-of-the-art open-source Multimodal Large Language Models (e.g., Qwen3-VL,
Qwen3.5, Llava, Kimi, and the GLM families) to these kinds of attacks.

Research Objectives