Letting Large Models Debate: The First Multilingual LLM Debate Competition

Current static evaluations and user-driven arenas have exhibited their limitations and biases in the previous year. Here, we explore a novel way to evaluate LLMs: debate. Debate is an excellent way to showcase reasoning strength and language abilities, used all across history, from the debates in the Athenian Ecclesia in the 5th century BCE to today's World Universities Debating Championship. Do today's large language models exhibit debate skills similar to humans? Which model is currently the best at debating? What can we learn from models when they debate against one another? To answer this question, BAAI has created a "Debate Arena" , allowing large models to compete against each other. Currently, it supports debate competitions in English, Chinese, Arabic and Korean.

The advancement of multimodal and multilingual technologies has exposed the limitations of traditional static evaluation protocols in capturing LLMs’ performance in complex interactive scenarios. Inspired by OpenAI’s “AI Safety via Debate” framework—which emphasizes enhancing models’ reasoning and logic through multi-model interactions ( [1] )—BAAI’s FlagEval Debate platform introduces a dynamic evaluation methodology to address these limitations. Recent research has demonstrated the potential of multi-agent debates in improving models’ reasoning capabilities and factual accuracy. For example, studies have shown that multi-agent interactions can significantly enhance models’ consistency and accuracy in logical reasoning and factual judgments ( [2] ), while others have indicated that multi-model debates encourage models to generate more truthful and coherent responses ( [3] ). While existing platforms like LMSYS Chatbot Arena offer foundational settings for multi-model interactions, they present certain limitations in practical evaluation:

BAAI’s FlagEval Debate platform addresses these challenges by introducing genuine multi-model debates. Models engage in direct confrontations, showcasing their reasoning processes and depth. This allows evaluators to observe and compare differences in viewpoints, logical reasoning, and argumentation strategies among models. The platform’s multilingual support and real-time debugging capabilities enable users to study model strengths in realistic and interactive settings, ultimately providing more discriminative and in-depth evaluation results.

To enhance fairness and flexibility, FlagEval Debate offers a developer customization feature, allowing participating model teams to fine-tune parameters, strategies, and dialogue styles based on their models’ characteristics and task requirements. This capability enables developers to optimize their models’ performance in debates, showcasing strengths while identifying areas for improvement. The real-time feedback loop fosters continuous optimization, allowing models to stand out in competitive evaluations.

FlagEval Debate represents a significant advancement in LLM evaluation methodologies. By incorporating multilingual support, developer customization, and a dual evaluation system, it offers a robust framework for assessing models in interactive, real-world scenarios. Moving forward, BAAI is committed to refining this platform to foster innovation, enhance evaluation methodologies, and drive standardization in AI practices. The goal is to provide developers and users with a forward-looking evaluation ecosystem, accelerating the evolution and deployment of advanced large language models.

We addressed these issues by refining the system prompt requirements and adjusting the input format, though each model still exhibited unique error patterns. At the same time, the experiments highlighted substantial opportunities for model enhancement. Performance variations indicated potential improvements in reasoning chains, linguistic expressiveness, and adversarial strategies. We encourage model creators to use our debate-tuning services to refine and optimize their models’ debate performance!

FlagEval Debate supports Chinese, English, Korean, and Arabic, encompassing a diversity of writing systems and languages. This multilingual capability provides a platform for models to perform in cross-cultural contexts and tests their adaptability and communication effectiveness across diverse linguistic environments. This addresses the global demand for multilingual LLM evaluation.

FlagEval Debate employs a unique dual evaluation system combining expert reviews with user feedback, assessing models from both technical and experiential perspectives:

In Q3 2024, we conducted extensive experiments on the FlagEval Debate platform to evaluate the impact of multi-model debates on models’ logical reasoning and differentiated performance. The experiments yielded several critical insights: 1. Most current models can engage in debate Our experiments demonstrated that all participating models, including closed-source variants, could effectively engage in debate tasks. This indicates that models across different architectures and training paradigms possess the requisite capabilities for logical reasoning and interactive dialogue in multi-model settings. This broad applicability enhances the relevance of FlagEval Debate as a comprehensive evaluation platform. We have empirically observed that some small open-source models still face challenges maintaining coherence and staying on topic.