ArXiv Preprint - Holistic Analysis of Hallucination in GPT-4V(ision): Bias and Interference Challenges

In this episode we discuss Holistic Analysis of Hallucination in GPT-4V(ision): Bias and Interference Challenges by Chenhang Cui, Yiyang Zhou, Xinyu Yang, Shirley Wu, Linjun Zhang, James Zou, Huaxiu Yao. The study introduces the Bingo benchmark to analyze hallucination behavior in GPT-4V(ision), a model processing both visual and textual data. Hallucinations, categorized as either bias or interference, reveal that GPT-4V(ision) prefers Western-centric images and is sensitive to how questions and images are presented, with established mitigation strategies proving ineffective. The findings expose similar issues in other leading visual-language models, suggesting an industry-wide challenge that necessitates novel solutions.