Consilium: When Multiple LLMs Collaborate

Picture this: four AI experts sitting around a poker table, debating your toughest decisions in real-time. That's exactly what Consilium, the multi-LLM platform I built during the Gradio Agents & MCP Hackathon , does. It lets AI models discuss complex questions and reach consensus through structured debate.

The platform works both as a visual Gradio interface and as an MCP (Model Context Protocol) server that integrates directly with applications like Cline (Claude Desktop had issues as the timeout could not be adjusted). The core idea was always about LLMs reaching consensus through discussion; that's where the name Consilium came from. Later, other decision modes like majority voting and ranked choice were added to make the collaboration more sophisticated.

This wasn't my original hackathon idea. I initially wanted to build a simple MCP server to talk to my projects in RevenueCat. But I reconsidered when I realized a multi-LLM platform where these models discuss questions and return well-reasoned answers would be far more compelling.

The research agent accesses five sources: Web Search, Wikipedia, arXiv, GitHub, and SEC EDGAR. I built these tools on an extensible base class architecture for future expansion while focusing on freely embeddable resources.

The protocol's interaction patterns map directly to Consilium's architecture:

Building Consilium reinforced my conviction that AI's future lies not just in more powerful individual models but in systems enabling effective AI collaboration. As specialized smaller language models become more efficient and resource-friendly, I believe roundtables of task-specific SLMs with dedicated research agents may provide compelling alternatives to general-purpose large language models for many use cases.

The current model selection includes Mistral Large, DeepSeek-R1, Meta-Llama-3.3-70B, and QwQ-32B. While notable models like Claude Sonnet and OpenAI's o3 are absent, this reflected hackathon credit availability and sponsor award considerations rather than technical limitations.

For models supporting function calling, I integrated a dedicated research agent that appears as another roundtable participant. Rather than giving models direct web access, this agent approach provides visual clarity about external resource availability and ensures consistent access across all function-calling models.

I'm currently integrating Open Floor Protocol support to allow users to add any OFP-compliant agents to their roundtable discussions. You can follow this development at: https://huggingface.co/spaces/azettl/consilium_ofp

The timing turned out to be perfect. Shortly after the hackathon, Microsoft published their AI Diagnostic Orchestrator (MAI-DxO) , which is essentially an AI doctor panel with different roles like "Dr. Challenger Agent" that iteratively diagnose patients. In their setup with OpenAI o3, they correctly solved 85.5% of medical diagnosis benchmark cases, while practicing physicians achieved only 20% accuracy. This validates exactly what Consilium demonstrates: multiple AI perspectives collaborating can dramatically outperform individual analysis.

After settling on the concept, I needed something that worked as both an MCP server and an engaging Hugging Face space demo. Initially I considered using the standard Gradio Chat component, but I wanted my submission to stand out. The idea was to seat LLMs around a table in a boardroom with speech bubbles, which should capture the collaborative discussion while also making it visually engaging. As I did not manage to style a standard table nicely so it was actually recognized as a table, I went for a poker-style roundtable. This approach also let me submit to two hackathon tracks by building a custom Gradio component and MCP server.

The custom Gradio component became the heart of the submission; the poker-style roundtable where participants sit and display speech bubbles showing their responses, thinking status, and research activities immediately caught the eye of anyone visiting the space. The component development was remarkably smooth thanks to Gradio's excellent developer experience, though I did encounter one documentation gap around PyPI publishing that led to my first contribution to the Gradio project.