In the ever-evolving landscape of machine learning and high-performance computing, maintaining optimal performance while managing costs is a complex challenge. Predibase's Deployment Health Analytics are designed to give you real-time insights into the health of your deployments and track crucial metrics that help you balance performance and cost. See for yourself by signing up for our 30-day free trial.
Understanding the Deployment Health Analytics
The Deployment Health Analytics are your command center for monitoring the various aspects of your deployment’s performance. They provide a holistic view of how well your deployment handles incoming requests, generates responses, and scales resources. Here’s a breakdown of the metrics you’ll find in the Deployment Health Analytics and how they interact to ensure your deployment runs smoothly.
1. Volume of Requests Being Handles (per Second)
The volume of incoming requests is the primary driver of all other metrics. This metric reflects the number of requests your deployment receives over time. A surge in request volume puts pressure on the deployment, potentially affecting throughput, latency, and queue duration. The Deployment Health Analytics allow you to monitor these request volumes historically and in real-time, helping you identify trends that might indicate a need for additional resources. For example, if you increase the number of requests per second and throughput (tokens / s) decreases, you’ll generally want to add additional replicas or modify the autoscaling parameters to ensure there are always sufficient GPUs available to meet your throughput requirements.
2. Throughput (Tokens per Second)
Throughput measures how many tokens your deployment processes each second. It’s a direct indicator of the deployment’s ability to handle requests efficiently. As the volume of incoming requests increases, the deployment aims to maintain throughput by leveraging existing GPUs or adding new ones. The Deployment Health Analytics provide detailed throughput metrics, enabling you to assess how effectively your deployment is managing the workload.
3. LoRAX Inference Time
LoRAX Inference Time provides a crucial metric for monitoring the efficiency of your SLMs. This metric specifically measures the time it takes for an SLM to generate a response once a request has been processed, offering valuable insight into the model's performance. While it doesn't account for network latency, the LoRAX Inference Time allows you to focus on the pure computational efficiency of your models, helping you optimize and fine-tune deployments for faster, more responsive interactions.
4. Queue Duration
Queue duration tracks how long requests wait before being processed. Longer queue durations can signal that the deployment is under strain and may require additional resources. With the Deployment Health Analytics, you can observe queue duration trends and configure thresholds that suit your operational needs. For instance, when setting up your deployment you can define the number of pending (or in-progress) requests each active replica must handle before scaling up to an additional replica, as well as the duration after which your deployment will automatically scale down replicas if it no longer needs them. This customization lets you strike a balance between tolerating longer queue times (performance) and scaling up additional GPUs (cost).
5. Number of GPU Replicas
The number of GPU replicas reflects the deployment’s response to varying demand. As requests increase, the autoscaling feature in Predibase will spin up additional GPU replicas to manage the load. Conversely, when demand decreases, replicas will be spun down to save on costs since you are only billed per active GPU-second. You can even configure a deployment to scale down to zero replicas. The Deployment Health Analytics allow you to monitor these scaling thresholds, including the duration after which the deployment will automatically scale down replicas if they are no longer needed.
6. GPU Utilization
GPU utilization indicates how much of the available GPU compute capacity is being used. High utilization means the GPUs are working at full capacity, which is efficient but leaves little room for unexpected spikes. Low utilization for use cases that are in production, on the other hand, may suggest over-provisioning and inefficiencies. The Deployment Health Analytics provide real-time data on GPU utilization, helping you ensure that your resources are neither underutilized nor overburdened.
The Interplay of Metrics
The Deployment Health Analytics integrate these metrics to offer a comprehensive view of your deployment’s health. When the volume of incoming requests spikes, queue duration starts to rise as requests wait for processing. At this point, the deployment may scale up by adding more GPU replicas, depending on the thresholds you’ve set. As new replicas come online, they help reduce queue duration and improve throughput.
Conversely, if request volume decreases, replicas scale down as fewer requests are waiting in the queue, based on the duration settings you’ve configured, helping to optimize costs.
Customizing Your Autoscaling Strategy
One of Predibase’s most powerful features is the ability to simply customize your autoscaling strategy. You can define specific thresholds for when to spin up or scale down GPU replicas, allowing you to manage the trade-off between speed, queue time, and cost efficiency. This flexibility ensures that your deployment can adapt to fluctuating demands while keeping costs in check.
You can configure these settings in the UI during deployment setup or via the SDK.
Conclusion
Predibase’s Deployment Health Analytics are an essential tool for managing and optimizing the performance of your machine learning deployments. By providing real-time insights into key metrics such as request volume, throughput, queue duration, GPU replicas, and GPU utilization, it empowers you to make informed decisions and maintain a balance between performance and cost. With the ability to customize scaling thresholds, you can tailor your deployments to your specific needs and ensure they run smoothly and efficiently, no matter how your workload changes.
Explore the Deployment Health Analytics today with our 30-day free trial and take control of your deployment’s performance. With Predibase, you have the tools you need to keep your deployments running at their best.
