Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Implemented
-
None
-
None
-
None
Description
This is one of a series of improvements to optimize the performance stability of our storage subsystem, which suffers from a number of problems. The end result is that there are periodically write stalls during data ingestion, even though the ingestion speed is relatively low.
This improvement will deal with the following issues:
1. Bypass all queuing of disk writes during LSM flush and merge operations. Queuing (by BufferCache and IOManager) will cause serious problems to the fairness of disk writes. Thus, a small flush operation could be severely interfered with a large merge operation and would take a much longer time to finish.
2. Perform regular disk forces during flush and merge operations (16MB by default). This is very helpful to limit the I/O queue length of the file system and provides fairness to queries and other writers. This optimization has been implemented in most storage systems today, including Couchbase Server.
3. Optionally, add the support for rate limiting of disk writes to ensure the performance stability of queries. The user can configure the maximum disk write bandwidth for each dataset. This ensures that the system can provide stable performance for both queries and writes, even with large background merges.