https://fdalvi.github.io/tags/coling/ Recent content in Coling on Fahim Dalvi Hugo en Mon, 20 Jan 2025 13:00:00 +0300 https://fdalvi.github.io/blog/2025-01-20-aradice-benchmarks-for-arabic-dialects/ Mon, 20 Jan 2025 13:00:00 +0300 https://fdalvi.github.io/blog/2025-01-20-aradice-benchmarks-for-arabic-dialects/ <p>Excited to share that our paper <a href="https://aclanthology.org/2025.coling-main.283/">AraDiCE: Benchmarks for Dialectal and Cultural Capabilities in LLMs</a> has been accepted at <a href="https://coling2025.org/">COLING 2025</a> in Abu Dhabi.</p> <p>Arabic isn&rsquo;t just one language — it&rsquo;s a family of dialects that vary dramatically from region to region. Yet most LLM evaluations treat Arabic as a monolith, using only Modern Standard Arabic (MSA). This paper addresses that gap by introducing <strong>AraDiCE</strong>, a benchmark that evaluates LLMs on both dialectal and cultural dimensions. We evaluated several LLMs on these benchmarks and found an interesting pattern: Arabic-specific models like Fanar, Jais and AceGPT do outperform general multilingual models on dialectal tasks. But significant challenges remain — particularly in dialect identification, generation, and translation.</p>