CASA: Cross-Attention via Self-Attention for Efficient Vision-Language Fusion

Moritz Böhle^*, Amélie Royer^*, Juliette Marrie^*, Edouard Grave, Patrick Pérez    ^*Equal contribution.

CASA is a novel vision-language modeling techniques that builds on — and improves — cross-attention for multimodal fusion. Specifically, CASA layers inject visual tokens into a text stream by using image-to-text cross-attention while additionally enabling text-to-text self-interaction in the same layer within local attention windows. This simple modification to the cross-attention design substantially improves performance while retaining the computational efficiency of cross-attention.

We evaluate CASA on several standard vision-language benchmarks spanning a variety of tasks (visual question answering, document understanding, OCR, etc.), comparing to other fusion mechanisms such as token insertion and cross-attention. To showcase the benefits of CASA for streaming applications, we further finetune and employ our model for the task of live video captioning.

Learn more about CASA