HBase
  1. HBase
  2. HBASE-1261

Improvements to flushing, compacting, and splitting

    Details

      Description

      This issue concludes my task for GSoC 2009
      The idea is to make all futures from Bigtable paper and other ideas in this processes of to flushing, compacting and splitting.

      Some talk in IRC related to this issue creation:

      [22:37] <jgray2> bigtable has an optimization which, when flushing the memcache, will actually write out a new storefile that merges the existing storefile with the memcache
      [22:37] <jgray2> that way you don't have more than one storefile
      [22:37] <jgray2> which is a huge inefficiency

        Issue Links

        There are no Sub-Tasks for this issue.

          Activity

          Hide
          Jonathan Gray added a comment -

          Closing this issue as duplicate. We made the minor/major compactions outlined by erik.

          We also made other improvements with all the HBASE-1234 HBASE-1249 HBASE-880 etc etc for 0.20

          Any other plans to improve flushes/compactions/splits should be handled in new issues targeted at 0.20 and beyond

          Show
          Jonathan Gray added a comment - Closing this issue as duplicate. We made the minor/major compactions outlined by erik. We also made other improvements with all the HBASE-1234 HBASE-1249 HBASE-880 etc etc for 0.20 Any other plans to improve flushes/compactions/splits should be handled in new issues targeted at 0.20 and beyond
          Hide
          Billy Pearson added a comment -

          currently with timestamps being provided by the client we can not do #3 in a minor compaction/merge
          the reason is we might delete the wrong version if we are not aware of all version with all timestamps.
          also I thank last time I checked we are doing #2 in minor compaction/merge

          Show
          Billy Pearson added a comment - currently with timestamps being provided by the client we can not do #3 in a minor compaction/merge the reason is we might delete the wrong version if we are not aware of all version with all timestamps. also I thank last time I checked we are doing #2 in minor compaction/merge
          Hide
          Erik Holstad added a comment -

          wo types of compactions, minor and major.
          Tasks that can be performed during a compaction:
          1. Sorting the KeyValue from the different file
          2. Checking TTL
          3. Checking number of versions to keep
          4. Removing KeyValues that are effected by deletes
          5. Removing the deletes

          For a major compaction all of the above is done, so the question is what we
          should do for a minor.

          The thing is that when merging the different files into one list we need to sort
          the smallest entry in all the files to be able to tell which one to put in first
          ,which means that we already are doing most of the work. So adding 2,3 and 4
          will only add 3 extra checks, but will probably give better performance in the
          system in total, but needs to be tested. The deletes can't be removes for a non
          major compaction, so they will be kept in the new file, but will only apply to
          older files, which will speed up reading.

          Today when doing the merge we open up a scanner for all the n files that we want
          to merge and for every value we put into the new file we do n compares, so if
          the total number of inserts are m we get something like m*n compares since we
          are not using any of the information from the last compare, so we end up redoing
          a lot of work.

          I propose a TreeSet or a Heap structure so we get something like m*log.
          After that we just need to do some testing to see how much the extra compares
          will add and how big the read performance gain will be by enforcing the extra
          checks.

          Show
          Erik Holstad added a comment - wo types of compactions, minor and major. Tasks that can be performed during a compaction: 1. Sorting the KeyValue from the different file 2. Checking TTL 3. Checking number of versions to keep 4. Removing KeyValues that are effected by deletes 5. Removing the deletes For a major compaction all of the above is done, so the question is what we should do for a minor. The thing is that when merging the different files into one list we need to sort the smallest entry in all the files to be able to tell which one to put in first ,which means that we already are doing most of the work. So adding 2,3 and 4 will only add 3 extra checks, but will probably give better performance in the system in total, but needs to be tested. The deletes can't be removes for a non major compaction, so they will be kept in the new file, but will only apply to older files, which will speed up reading. Today when doing the merge we open up a scanner for all the n files that we want to merge and for every value we put into the new file we do n compares, so if the total number of inserts are m we get something like m*n compares since we are not using any of the information from the last compare, so we end up redoing a lot of work. I propose a TreeSet or a Heap structure so we get something like m*log . After that we just need to do some testing to see how much the extra compares will add and how big the read performance gain will be by enforcing the extra checks.
          Hide
          stack added a comment -
          Show
          stack added a comment - I added this issue up here: http://wiki.apache.org/general/SummerOfCode2009

            People

            • Assignee:
              Unassigned
              Reporter:
              Evgeny Ryabitskiy
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development