At Hostna, we run a variety of virtualized environments where performance consistency is the primary metric for success. When your application experiences unexplained latency despite low internal resource usage, the culprit is often found in the relationship between your virtual machine and the physical host. In the world of virtualization, specifically within a Kernel-based Virtual Machine (KVM) environment, the hardware resources are shared among multiple tenants. If one tenant consumes an excessive amount of cycles, it can impact the availability of the physical CPU for others. This phenomenon is known as the noisy neighbor effect, and it is measured through a specific metric: CPU steal time.
CPU steal time, represented as st in most Linux monitoring tools, is the percentage of time a virtual CPU (vCPU) waits for a physical CPU while the hypervisor is busy servicing another virtual machine. It is involuntary wait time. Unlike iowait, which indicates a delay in storage or network I/O, steal time indicates that your server is ready to run but is being held back by the hypervisor scheduler. Understanding the mechanism behind this metric is essential for anyone managing a high-performance virtual servers environment where timing-sensitive tasks are common.
| Symptom | Probable Cause | Recommended Fix |
|---|---|---|
High st in top (over 10%) |
Physical CPU oversubscription or noisy neighbor. | Migrate to a plan with dedicated resource allocation. |
High wa and st together |
Host-level storage contention (NVMe IOPS saturation). | Verify if RAID10 versus ZFS overhead is impacting the host. |
| Intermittent latency spikes | Hypervisor maintenance or temporary batch processing. | Monitor st patterns using sar or Prometheus. |
The Mechanism of CPU Steal in KVM
To understand why steal time occurs, we must look at how the Linux kernel handles timekeeping. In a physical server, the kernel expects to have full access to the CPU ticks. In a virtualized Linux environment, the hypervisor (the host) manages the scheduling of multiple guest operating systems. When the host schedules a different guest on the physical core assigned to your VPS, your guest kernel is essentially paused. However, the guest kernel’s internal clock continues to track the passage of real time. When the guest is resumed, it compares its internal clock to the host’s pvclock. The difference between the time the guest wanted to run and the time it actually ran is recorded as st.
On the Hostna platform, we prioritize scheduling efficiency to minimize this overhead. However, in any shared environment, the hypervisor must balance requests. If you are running a database like MariaDB and notice that query execution time has doubled while the us (user) and sy (system) percentages remain low, check the st column. A high steal value means your database is waiting for the processor, which can lead to a backlog of connections and eventual exhaustion of the pm.max_children limit in PHP-FPM or connection limits in SQL. You can find more about tuning these variables in the performance tuning documentation provided by the MariaDB community.
Diagnostic Commands for Identifying Contention
The most common tool for a quick check is top. When you run top, look at the third line, which starts with %Cpu(s):. The value labeled st is your steal time. If this value is consistently above 5 percent, your workload is being impacted. If it reaches 10 percent or higher, the impact on application performance will be severe. To get a better view of how this changes over time, use vmstat, which provides a summary of system activity.
vmstat 1 10This command will report system statistics every second for ten iterations. The very last column, st, shows the steal time. If you see zeroes, your neighbor is not currently impacting you. If you see fluctuating numbers, it indicates a bursty neighbor. For a per-core breakdown, which is useful on multi-core systems, use mpstat from the sysstat package:
mpstat -P ALL 1This command is particularly useful because it can reveal if the steal time is isolated to a specific vCPU. If one core shows 20 percent steal while others show 0 percent, it suggests the hypervisor is struggling to schedule that specific virtual thread, possibly due to the physical core being pinned by another process on the host.
Correlation with I/O Wait and Load Average
It is a common mistake to confuse st with wa (iowait). While both represent a CPU that is not doing productive work, their causes are different. wa means the CPU is waiting for a response from a disk or network device. In modern German-hosted infrastructure, where NVMe drives are standard, wa should be near zero. If you see high wa, you should investigate NVMe IOPS versus throughput limits. However, if st is high, the disk is not the problem; the scheduler is.
Another indicator is the load average. In Linux, the load average includes processes in the R (running) and D (uninterruptible sleep) states. A process waiting for the CPU due to steal time is still in the R state. Therefore, you might see a load average of 4.0 on a 2-core machine, but top shows only 10 percent us usage. If st is high, that extra load is literally the weight of the neighbors’ tasks preventing your own from completing.
When to Move from vCPU to Dedicated Cores
Most VPS instances use vCPU, which are shared time-slices of physical cores. This is efficient for bursty workloads like web servers but poor for sustained tasks like video encoding or high-traffic databases. If your st metrics are consistently high, it is a sign that your workload has outgrown the shared nature of the plan you are on. Transitioning to a Virtual Dedicated Server (VDS) provides you with dedicated physical cores, effectively eliminating st because no other tenant can schedule tasks on your assigned silicon.
When evaluating performance, consider the innodb_buffer_pool_size. If your database is struggling and st is high, increasing the buffer pool won’t help because the CPU cannot process the data already in RAM. In this case, the bottleneck is the compute availability, not the memory. A 4 GB machine with dedicated cores will often outperform an 8 GB machine with high steal time. This is why checking our service level agreements and understanding the resource guarantees of your specific tier is vital for long-term stability.
Monitoring and Automated Alerting
Manual checks with top are fine for immediate troubleshooting, but production environments require automated monitoring. If you use Prometheus, the node_exporter provides a metric called node_cpu_seconds_total with a mode="steal" label. You can calculate the rate of steal time with a query like:
rate(node_cpu_seconds_total{mode="steal"}[5m]) * 100Setting an alert for when this value exceeds 10 percent for more than 15 minutes is a sensible default. This prevents false positives from temporary host-side spikes (like backups or hypervisor updates) while catching persistent noisy neighbor issues. If you are unsure how to implement this, our technical documentation contains further examples for various monitoring stacks.
If you have identified sustained steal time and your workload is optimized, the next step is to analyze the host environment. Sometimes, a simple migration to a different physical node within the same data center can resolve the issue if that specific node has a particularly “loud” neighbor. However, for critical production workloads, the only way to ensure 0 percent steal time is to move to hardware where the cores are not shared.
Conclusion and Support Steps
Identifying noisy neighbor issues through CPU steal time is a matter of looking at the right metrics in top, vmstat, and mpstat. By correlating st with load average and iowait, you can distinguish between a code-level bottleneck and a platform-level resource contention. If your st values are consistently high, it is time to evaluate your resource needs and consider moving to a tier with dedicated cores. If you continue to see performance issues after reviewing these metrics, please contact our support team through the dashboard. Provide the output of vmstat 1 60 and a screenshot of your top output sorted by CPU usage so we can investigate the host-side metrics and ensure your server is performing as expected.
This article was drafted with AI assistance from Hostna’s own research pipeline and published without individual review. Tell us if anything here is wrong.