We are seeing some issues with iowait. On a machine with 16 cores and 60 GB memory and SSD (max IO around 250 mb/s) (virtual machine on google) we see around constantly 20% iowait on top and iotop shows its the mongoDB process. Only around 10MB/s are actually read from the disk and as I said, we know from other tests that the machine can do much much more.
The query producing this pattern sorts a result from a query by an attribute (all indexed). After running queries like this a few times it works better. The total index size is around 4GB, so I assume it gets loaded in memory directly.
Is there any suggestions on how to debug this? I cant see how there can be such a high iowait when all computing resources are not even barely utilized.
Note: Running MongoDB 3.0.10 on Ubuntu 14.04