🚀 Accelerating LLM Inference with TGI on Intel Gaudi

We're excited to announce the native integration of Intel Gaudi hardware support directly into Text Generation Inference (TGI), our production-ready serving solution for Large Language Models (LLMs). This integration brings the power of Intel's specialized AI accelerators to our high-performance inference stack, enabling more deployment options for the open-source AI community 🎉

We've fully integrated Gaudi support into TGI's main codebase in PR #3091 . Previously, we maintained a separate fork for Gaudi devices at tgi-gaudi . This was cumbersome for users and prevented us from supporting the latest TGI features at launch. Now using the new TGI multi-backend architecture , we support Gaudi directly on TGI – no more finicking on a custom repository 🙌

This integration supports Intel's full line of Gaudi hardware :

We invite the community to try out TGI on Gaudi hardware and provide feedback. The full documentation is available in the TGI Gaudi backend documentation . 📚 If you're interested in contributing, check out our contribution guidelines or open an issue with your feedback on GitHub. 🤝 By bringing Intel Gaudi support directly into TGI, we're continuing our mission to provide flexible, efficient, and production-ready tools for deploying LLMs. We're excited to see what you'll build with this new capability! 🎉

You can also find more information on Gaudi hardware on Intel's Gaudi product page

The easiest way to run TGI on Gaudi is to use our official Docker image. You need to run the image on a Gaudi hardware machine. Here's a basic example to get you started:

For comprehensive documentation on using TGI with Gaudi, including how-to guides and advanced configurations, refer to the new dedicated Gaudi backend documentation .