Running large AI models is expensive, especially when thousands of requests compete for computing resources. The open-source llm-d project aims to tackle that problem with version 1.0, introducing new capabilities for managing AI inference across distributed infrastructure. Founded by Red Hat, IBM, Google Cloud, CoreWeave, and NVIDIA, the project is now governed by the Cloud Native Computing Foundation (CNCF). For eeNews Europe readers, the release is particularly relevant to companies developing AI infrastructure and deploying large language models. It offers ways to improve GPU utilization, manage demanding workloads, and reduce the cost of running AI applications. AI Inference Gets More Efficient One of the biggest challenges in AI inference is managing the information models need to process and reuse. This becomes especially demanding with AI agents, which often handle long conversations, access large amounts of context, and run multiple tasks simultaneously. llm-d addresses these challenges through techniques such as cache-aware routing, memory offloading, and separating the processing of input prompts from generating responses. Version 1.0 adds improved workload scheduling, allowing different users and applications to share computing resources without overwhelming the system. It also introduces high-availability routing and coordinated updates, helping operators maintain services during maintenance. Lower Costs on Existing GPUs Recent demonstrations show what these improvements could mean for large AI deployments. In one benchmark, researchers used llm-d to run GLM 5.2, an open-weight model with approximately 753 billion parameters, on NVIDIA GPUs. The system delivered more than 6.6 million output tokens per minute at peak and supported up to 3,000 concurrent coding agents without preemptions. Based on current cloud rental rates, the researchers estimated inference costs five to ten times lower per token than comparable commercial API pricing for the tested workloads. Another project at IBM Research used open models and llm-d to investigate quantum error correction hardware. Researchers reported inference costs 11.5 times lower than an entirely proprietary configuration, with a cache hit rate of approximately 98%. These results come from specific workloads and configurations, rather than general performance guarantees. Built for Production AI Workloads Beyond performance, llm-d 1.0 focuses on making distributed AI inference easier to operate in production environments. The release introduces multi-tenant quality of service, session-aware routing, and support for updates without planned downtime. The project now includes more than 500 contributors from over 25 organizations. “As organizations move beyond AI experimentation and into full-scale production, the bottleneck shifts from generating answers to managing the economics, operational reliability, and extreme infrastructure demands of complex agentic workloads,” said Brian Stevens, senior vice president and AI chief technology officer at Red Hat. Google Cloud is also integrating llm-d 1.0 into Google Kubernetes Engine, with support across its TPU and NVIDIA GPU portfolio. The release gives organizations another open source option for managing large AI workloads while making better use of existing computing infrastructure. The post llm-d 1.0 Targets Lower AI Inference Costs appeared first on eeNews Europe .