Uploaded image for project: 'Accumulo'
  1. Accumulo
  2. ACCUMULO-1787

support two tier compression codec configuration

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.0
    • Component/s: None
    • Labels:
      None

      Description

      Given our current configuration of one compression codec per table we have the option of leaning towards performance with something like snappy or leaning towards smaller footprint with something like gzip. With a change to the way we configure codecs we might be able to approach the best of both worlds. Consider the difference between files that have been written by major or minor compactions and files that exist at any given point in time. For better footprint on disk we care about the latter, but for total CPU usage over time we care about the former. The two distributions are distinct because Accumulo deletes files after major compactions. If we figure out whether a file is going to be long-lived at the time we write it then we can pick the compression codec that optimizes the relevant concern.

      One way to distinguish is by file size. Accumulo writes many small files and later major compacts those away, so the distribution of written files is skewed towards smaller files while the distribution of files existing at any point in time is skewed towards larger files. I recommend for each table we support a general compression codec and a second codec for files under a configurable size.

        Attachments

        1. hybrid2.diff
          3 kB
          Chris McCubbin
        2. AccumuloWatcher.java
          3 kB
          Chris McCubbin
        3. hybrid.diff
          7 kB
          Chris McCubbin
        4. ci_file_sizes.png
          5 kB
          Adam Fuchs

          Issue Links

            Activity

              People

              • Assignee:
                milleruntime Michael Miller
                Reporter:
                afuchs Adam Fuchs
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 3h
                  3h