Opening article
Hugging Face’s Infrastructure team has built a robust monitoring and alerting system that supports its production environment. The system includes three key alerts that monitor network traffic, cost‑optimizing traffic paths, and log pipeline health. The team also runs a sophisticated logging pipeline using Filebeat, Logstash, Elasticsearch, and AWS Glue to collect, process, and archive logs for operational and analytical use.
The Infrastructure team at Hugging Face is excited to share a behind-the-scenes look at the inner workings of Hugging Face's production infrastructure, which we’ve had the privilege of helping to build and maintain. Our team's dedication to designing and implementing a robust monitoring and alerting system has been instrumental in ensuring the stability and scalability of our platforms. We’re constantly reminded of the impact that our alerts have on our ability to identify and respond to potential issues before they become major incidents.
In this blog post, we’ll dive into the details of three mighty alerts that play their unique role in supporting our production infrastructure, and explore how they've helped us maintain the high level of performance and uptime that our community relies on.
Oftentimes, with cloud computing architectures, where data flows between private and public networks, implementing a NAT (Network Address Translation) gateway stands as a steadfast best practice. This gateway acts as a strategic gatekeeper, monitoring and facilitating all outbound traffic towards the public Internet. By centralizing egress traffic, the NAT gateway offers a strategic vantage point for comprehensive visibility. Our team can easily query and analyze this traffic, making it an invaluable asset when working through security, cost optimization or various other investigative scenarios.
Cost optimization is a critical aspect of cloud infrastructure management, and understanding the pricing dynamics is key. In data centers, pricing structures often differentiate between east-west traffic (typically communication within the same rack or building) and north-south traffic (communication between further away private networks or the internet). By monitoring network traffic volume, Hugging Face gains valuable insights into these traffic patterns. This awareness allows us to make informed decisions regarding infrastructure configuration and architecture, ensuring we limit incurring needless costs.
To manage the lifecycle of logs within Elasticsearch, we employ a robust storage and lifecycle management strategy. This ensures that logs are retained in Elasticsearch for a defined period, providing quick access for operational and troubleshooting purposes. After this retention period, logs are offloaded to long-term archival storage. The archival process involves an automated tool that reads logs from Elasticsearch indexes, formats them as Parquet files—an efficient columnar storage format—and writes them to our object storage system.
Archival jobs, responsible for transferring logs from Elasticsearch to object storage, have also encountered challenges. On occasion, these jobs can be resource-intensive, with their performance becoming sensitive to node size and memory availability. In cases where junk data or unusually large log entries pass through the pipeline, they can strain the archival process, leading to failures due to memory exhaustion or node capacity limits. This underscores the importance of data validation and filtering earlier in the pipeline to prevent such issues from reaching the archival stage.
One of our key alerts is designed to notify us when our network traffic volume surpasses a predefined threshold. This alert serves multiple purposes. Firstly, it acts as an early warning system, alerting us to any unusual spikes in traffic that might indicate potential issues or unexpected behavior. Secondly, it prompts us to regularly review our traffic trends, ensuring we stay on top of our infrastructure's growth and evolving needs. This alert is set at a static threshold, which we have fine-tuned over time, ensuring it remains relevant and effective. When triggered, it often coincides with periods of refactoring our infrastructure.
For instance, when integrating third-party security and autoscaling tools, we've observed increased telemetry data egress from our nodes, triggering the alert and prompting us to optimize our configurations.
On another occasion, adjustments to our infrastructure lead to mistakenly avoiding a private, low-cost path between product-specific infrastructure (ex. traffic destined to the Hub from a Space to interact with repository data). To elaborate further, the most impactful workloads in terms of cost savings we’ve found are those that access object storage. Fetching objects directly prices cheaper than going through CDN-hosted assets for our LFS repository storage and additionally does not require the same security measures that our WAF provides compared to public requests arriving at our front door. Leveraging DNS overrides to switch traffic through private network paths and public network paths has become a valuable technique for us, driven by the CDKTF AWS provider .
Robust monitoring and efficient log management are critical for AI platforms that handle large volumes of data and require high uptime. These practices help maintain performance, reduce costs, and ensure reliable operations.