Researchers from Apple conducted a comprehensive study on preference alignment in Multimodal Large Language Models (MLLMs), which face challenges like hallucination, including producing responses inconsistent with image content. The study independently analyzes each aspect of preference alignment, categorizing algorithms into offline methods (such as Direct Preference Optimization) and online methods (such as online-DPO), and shows that combining offline and online methods can improve model performance in certain scenarios. The authors reviewed various published multimodal preference datasets and discussed how their construction details impact model performance. Based on these insights, they introduced a novel method for creating multimodal preference data called Bias-Driven Hallucination Sampling (BDHS), which requires neither additional annotation nor external models. They demonstrated that BDHS can achieve competitive performance to previously published alignment work for multimodal models across a range of benchmarks. The paper also notes that off-the-shelf MLLMs demonstrate powerful inherent modality alignment properties, despite a substantial modality gap in Contrastive Language-Image Pretraining (CLIP) feature space.