Introducing GhazalBench for Evaluating LLMs in Persian

Researchers have introduced a new tool called GhazalBench, developed to assess large language models (LLMs) in their interaction with Persian poetry, specifically ghazals. This benchmark evaluates two main abilities: the production of accurate prose paraphrases of couplets and the retrieval of canonical verses based on varying prompts. While some LLMs can understand poetic meaning well, they often struggle with recalling exact verses, particularly in completion tasks, a problem less pronounced in recognition tasks. The implications of GhazalBench are significant for the evaluation of LLMs that engage with culturally specific texts. The observed dissociation in model performance suggests that current language models may be limited by their training exposure rather than their architecture. This research underscores the importance of creating evaluation frameworks that can effectively assess both the meaning and form of culturally significant texts, opening avenues for more nuanced and effective AI applications in language processing.