Papers with Code has relaunched its search engine using a hybrid search architecture designed to make AI research more accessible. According to Hugging Face, the system integrates PostgreSQL full-text search for exact keyword matching with pgvector for semantic recall, combining both methodologies through the reciprocal rank fusion algorithm. The platform currently maintains embeddings for over 110,000 current papers sourced from arXiv and Daily Papers. To optimize performance and resource utilization, the architecture separates the expensive offline corpus building from the online search service. The throughput-heavy embedding pipeline runs as Hugging Face Jobs on NVIDIA L4 GPUs, processing data streamed from a repeatable-read PostgreSQL snapshot and storing durable artifacts in private Storage Buckets. This ensures that retries can resume work without overwriting existing files. On the online request path, a pinned model version of Qwen3-Embedding-0.6B is deployed as an authenticated Inference Endpoint backed by Text Embeddings Inference. To maintain high availability during cold starts or high traffic, the client query system features a strict fallback mechanism that immediately skips the semantic branch and relies on lexical results if the endpoint is scaling up or times out. This split design allows the platform to maintain fast lookup latencies while scaling down to zero replicas when idle to save costs.
How Hugging Face Inference Endpoints, Jobs, and Buckets Power Search on Papers with Code
Papers with Code has revived its search engine using a hybrid architecture that combines keyword and vector search. Built with Hugging Face Inference Endpoints, Jobs, and Buckets, the system scales efficiently while handling more than 110,000 research papers sourced from arXiv and Daily Papers. The setup splits resource-intensive batch embedding generation from lightweight online request routing to ensure fast and resilient retrieval.
Why it matters
This architecture demonstrates a practical blueprint for combining batch processing and real-time inference to scale AI-driven research search engines efficiently and reliably.