My experience with multi tenant environments is that when there is a performance issue it is a 90% risk that it is caused by poor disk performance. This comes as no surprise since hard disk performance is about the same as it was 10 years ago, but everything else in the infrastructure is way faster than 10 years ago.
There are three ways to measure disk performance,
- I/O Per Second (IOPS)
- Throughput in MegaBytes per second (MB/s)
- Latency in milliseconds (ms)
Many just measure performance in throughput which is ok, if you only have large sequential I/O, but that is often not the case. IOPS is often a more relevant way to measure performance and databases, since they often require a lot of IOPS. It is not as easy to measure IOPS as it depends on the record size and how the data is accessed. The latency is not measured very often but my experience is that high latency is often the first sign of a performance problem and this will make systems with low I/O loads slow as well.
A high performance hard drive has an average latency of 4ms and a general rule is that a storage system should have a latency of less than 10ms to be considered ok in performance. The problem is when the storage system is heavily loaded and another server (the noisy neighbor) is using the same disk and your IO is queued then you can get latency numbers up to 20-30ms. A fast hard drive can deliver about 200 IOPS but all disksystems today have cache to be able to handle more IOPS and reduce latency. The problem is that you cannot fit all data in the cache and then you cannot get predictable high performance.
Solid State Drives (SSD) are great, as they do not have the latency issues as traditional hard drives do and they can deliver a lot more IOPS.
In this video we will show how you can benchmark your disk system and also show how a platform with SSD will give you predictable high performance.