Abstract

Recent advances in multimodal Vision–Language Models (VLMs) have introduced new architectural mechanisms designed to strengthen fine-grained visual-textual alignment. Among these, the DeepStack architecture (see [1]), adopted for example in state-of-the-art models such as Qwen3-VL, represents a significant shift in how visual features are injected into the language model. Instead of feeding all visual tokens at a single entry point, DeepStack extracts intermediate features from multiple layers of the vision encoder and reinjects them across several depths of the LLM. Intuitively, this creates a layered fusion pipeline in which visual information is repeatedly refreshed and integrated, improving multimodal reasoning while fundamentally altering internal joint text and image embedding space.

In parallel, research in adversarial machine learning has shown that many white-box attacks operate directly in the text-vision joint embedding space. Techniques such as suppressing refusal directions (see, [2]), steering semantic directions to induce hallucinations (see, e.g., [3, 4]); applying PGD on visual inputs have proven effective on architectures with a single fusion point. However, DeepStack disrupts these assumptions: perturbations applied at one level may be overridden by later visual feature injections, making such attacks substantially harder to implement efficiently and less stable.

This thesis aims to systematically analyze DeepStack-based architectures and to investigate whether effective and computationally feasible white-box multilayer attacks can still be constructed. The project will explore multi-level PGD, coordinated perturbations of intermediate vision encoder features, and manipulation of multi-stage visual-textual fusion points. The broader goal is to evaluate the vulnerability and robustness of DeepStack-enabled models by comparing them to architectures that rely on traditional single-point visual fusion.

Research Objectives

• Architectural Analysis: Characterize DeepStack’s multi-layer visual feature injection and identify critical fusion points in open source achitectures
• Attack Development: Design and implement multilayer white-box adversarial attacks (e.g., multi-level PGD, embedding steering) capable of influencing internal activations despite DeepStack’s structure.
• Benchmarking: Compare the robustness of DeepStack-based VLMs against non- DeepStack models through quantitative adversarial evaluation.

Expected Outcomes

• Prototype implementations of multilayer adversarial attacks tailored to DeepStack architectures.
• Experimental evaluation comparing DeepStack to traditional VLM fusion designs.
• Insights into the security and robustness implications of layered multimodal fusion.

Requirements and Technologies

• Proficiency in Python and the PyTorch / HuggingFace ecosystem.
• Background in deep learning, transformers, and adversarial machine learning.
• Interest in VLM robustness, model security, and multimodal architectures.

References

1. Lingchen Meng et al. “DeepStack: Deeply Stacking Visual Tokens is Surprisingly Simple and Effective for LMMs”. In: The Thirty-eighth Annual Conference on Neural Information Processing Systems. 2024. url: https://openreview.net/forum?id=fXDpDzHTDV.
2. Andy Arditi et al. “Refusal in language models is mediated by a single direction”. In: Advances in Neural Information Processing Systems 37 (2024), pp. 136037–136083.
3. Eugene Bagdasaryan et al. “Adversarial illusions in Multi-Modal embeddings”. In: 33rd USENIX Security Symposium (USENIX Security 24). 2024, pp. 3009–3025.
4. Haobo Wang et al. SGHA-Attack: Semantic-Guided Hierarchical Alignment for Transferable Targeted Attacks on Vision-Language Models. 2026. arXiv: 2602.01574 [cs.CV]. url: https://arxiv.org/abs/2602.01574.