Orca Security and AWS collaborated to build Discovery AI, a natural‑language query agent that converts plain text questions into Sonar queries for over 900 cloud infrastructure models. The solution, deployed on Amazon Bedrock, uses a nine‑stage pipeline that combines retrieval‑augmented generation, model selection, and continuous evaluation. Discovery AI achieves significant performance gains, reducing query costs by 90%, input token usage by 93%, and latency by 75% compared to the initial baseline. The architecture employs top‑20 nearest‑neighbor retrieval with Amazon Titan embeddings and reranks results with Cohere Rerank 3.5 to provide the generator with diverse, relevant examples. A model selector running on Anthropic Claude Haiku 4.5 limits schema exposure to only the models needed for a given query, improving cost and accuracy. Continuous evaluation harnesses a curated dataset and an LLM‑as‑judge scoring layer to monitor changes, while an internal labeling UI allows domain experts to correct outputs and feed new examples back into the system in real time. This approach turns Discovery AI into a living system that adapts to evolving customer environments and new question patterns, maintaining high accuracy without redeployments. The project showcases how combining LLMs, retrieval, and iterative feedback loops can deliver efficient, scalable natural‑language interfaces for complex security data.
Natural language security queries at Orca Security using Amazon Bedrock
Orca Security partnered with AWS to create Discovery AI, a natural‑language query agent built on Amazon Bedrock that translates plain text into Sonar queries for over 900 cloud models. The nine‑stage pipeline uses retrieval‑augmented generation, model selection, and continuous evaluation to reduce query costs by 90%, input tokens by 93%, and latency by 75% compared to the baseline. The system incorporates top‑20 nearest‑neighbor retrieval with Amazon Titan embeddings and Cohere Rerank 3.5, and employs a labeling UI to continuously improve accuracy.
Why it matters
By enabling fast, cost‑effective natural‑language queries over complex cloud security data, Discovery AI demonstrates how LLMs and retrieval techniques can streamline operations in security tooling, a key area for AI adoption in enterprise IT.