Uploaded image for project: 'James Server'
  1. James Server
  2. JAMES-2925

Decreasing chunk_length_in_kb for read heavy workloads

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Abandoned
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: cassandra, mailbox
    • Labels:
      None

      Description

      James mostly serve primarily reads with a ration often other 80%.

      We often benefit from read optimization.

      Such an optimization is the size of chunks being LZ4 compressed within SSTables files:

      • bigger chunks means better compression
      • but also means all reads need to load the full chunk size, thus costing IOs

      For read heavy workloads, experiment shows decreasing the chunk size from 64KB (default) to a more reasonable value (like 4KB) often leads to significant performance enhancement.

      As an adoption step, we should conduct a performance test on read-heavy mailbox-metadata. If conclusive, we can consider adoption on some other often read metadata.

      As a reference here is a reference article on the last pickle blog: https://thelastpickle.com/blog/2018/08/08/compression_performance.html

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              btellier Benoit Tellier
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: