Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-3871

Speedup LoadIncrementalHFiles by parallelizing HFile splitting

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.90.2
    • 0.92.0
    • mapreduce
    • None
    • Reviewed

    Description

      From Adam w.r.t. HFile splitting:
      There's actually a good number of messages of that type (HFile no longer fits inside a single region), unfortunately I didn't take a timestamp on just when I was running with the patched jars vs the regular ones, however from the logs I can say that this is occurring fairly regularly on this system. The cluster I tested this on is our backup cluster, the mapreduce jobs on our production cluster output HFiles which are copied to the backup and then loaded into HBase on both. Since the regions may be somewhat different on the backup cluster I would expect it to have to split somewhat regularly.

      This JIRA complements HBASE-3721 by parallelizing HFile splitting which is done in the main thread.

      Attachments

        1. 3871.patch
          4 kB
          Ted Yu

        Activity

          People

            yuzhihong@gmail.com Ted Yu
            yuzhihong@gmail.com Ted Yu
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: