Presenters

Source

Scaling LLMs: A Kubernetes Deep Dive 🚀💡👨‍💻

Deploying Large Language Models (LLMs) in production is no longer a simple matter of scaling web applications. It demands a fundamentally new approach, and Kubernetes is emerging as the central orchestration platform. Let’s dive into the latest techniques and innovations for mastering LLM deployments!

1. Deploying Small LLMs on Kubernetes: A Balancing Act 🛠️

Getting started with smaller LLMs (around 1 billion parameters) on Kubernetes can be surprisingly tricky. A recent presentation highlighted a practical approach using the VLM framework, demonstrating how to leverage Kubernetes for container orchestration and kubectl for management.

The Initial Hurdles:

  • Slow Downloads: Automatic model weight downloads from Hugging Face initially took a whopping 18 seconds!
  • Resource Constraints: Scaling proved difficult due to limited resources within the Kubernetes cluster.

Optimization Roadmap:

To overcome these challenges, the team outlined a comprehensive plan:

  • Local Model Weight Caching: The biggest win – eliminating repeated downloads!
  • Kubernetes Resource Optimization: Ensuring adequate CPU, memory, and GPU resources.
  • Model Quantization: Reducing model size and accelerating inference.
  • Load Balancing: Distributing traffic for scalability and resilience.
  • Automated Deployment (CI/CD): Streamlining the deployment process.
  • VLM-Specific Optimizations: Leveraging the VLM framework’s unique capabilities.
  • Inference Process Profiling: Pinpointing and optimizing performance bottlenecks.

2. Revolutionizing LLM Deployments with Kubernetes Gateway API 🌐

The traditional Kubernetes Ingress API is reaching its limits. To address this, a new architecture is emerging that combines the Kubernetes Gateway API with a custom Inference Extension.

Why the Change?

The legacy Ingress API is essentially in maintenance mode, lacking the flexibility needed for dynamic LLM routing across namespaces.

The Dynamic Duo: Gateway API + Inference Extension

This new architecture introduces:

  • GatewayClass: Defines the load balancer type (HTTP, TCP, UDP).
  • Gateway: Represents the actual load balancer instance.
  • HTTPRoute: Defines routing rules mapping a Gateway to a Kubernetes Service.
  • InferencePool: Groups LLM model server pods (e.g., “Gemma 1” and “Llama 3”) for simplified management and scaling.
  • EndpointPicker: Intelligently routes traffic based on model server health and performance metrics.

Key Benefits:

  • Model-Aware Routing (BBR): Route requests based on the specified model (e.g., /v1/completions?model=llama3).
  • Serving Priority: Prioritize traffic for critical applications or users.
  • Dynamic Failover/Failback: Automatically handle model server unavailability.
  • Model Rollouts: Gradually roll out new model versions.

3. LLMD: Discreted Serving for Unprecedented Scaling 💾

LLMD (Discreted Serving) tackles the unique scaling challenges of LLMs that traditional web application scaling methods simply can’t handle. The core innovation? Separating the inference process into distinct phases: prefill (intensive prompt processing) and decode (response generation).

Why Traditional Scaling Fails:

LLMs demand significantly more GPU memory and compute power, rendering standard autoscaling ineffective.

LLMD’s Solution:

  • Independent Scaling: Allocate resources specifically to prefill and decode phases.
  • Hardware Optimization: Use A100 GPUs for prefill and L4 GPUs for decode.
  • Configurable Thresholds: Dynamically switch between full and streamlined versions based on prompt length.
  • Serving Profiles: Easily switch between “discreted” and standard serving profiles.
  • Multi-Node Serving: Distribute models across multiple nodes.
  • GPU KV Caching Transfer: Efficiently move key-value caches between GPUs.

4. Kubernetes Takes the Helm: Orchestrating Large-Scale LLM Deployments 📡

Deploying massive LLMs, like DeepSeek’s quantized model (1,370 memory across 52 A4 GPUs), requires sophisticated orchestration. Kubernetes, combined with innovative tools, is stepping up to the challenge.

Key Technologies & Strategies:

  • Leader-Worker Sets: Efficiently manage workloads across multiple nodes.
  • Project Kueue: Open-source project for intelligent batching and resource sharing on Kubernetes.
  • Dynamic Resource Allocation (DRA): (Upcoming feature) Simplify resource claiming and IP address management.
  • LoRA Adapters: Lightweight personalization layers for tasks like translation, reducing the need for full model retraining.
  • GPU KV caching transfer technology: Facilitates efficient communication between GPUs.

The Future is Bright! ✨

The speakers emphasized the importance of community feedback and will be sharing code and presentation slides on GitHub. By embracing these new techniques and tools, we can unlock the full potential of LLMs and bring them to a wider audience.

Appendix