Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-13985

Add configuration to skip validating HFile format when bulk loading

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 0.98.13
    • Fix Version/s: 0.98.14, 1.2.0, 1.3.0, 2.0.0
    • Component/s: None
    • Labels:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      A new config, hbase.loadincremental.validate.hfile , is introduced - default to true
      When set to false, checking hfile format is skipped during bulkloading.
      Show
      A new config, hbase.loadincremental.validate.hfile , is introduced - default to true When set to false, checking hfile format is skipped during bulkloading.

      Description

      When bulk loading millions of HFile into one HTable, checking HFile format is the most time-consuming phase. Maybe we could use a parallel mechanism to increase the speed, but when it comes to millions of HFiles, it may still cost dozens of minutes. So I think it's necessary to add an option for advanced user to bulkload without checking HFile format at all.
      Of course, the default value of this option should be true.

        Attachments

        1. HBASE-13985.patch
          4 kB
          Victor Xu
        2. HBASE-13985-v2.patch
          4 kB
          Victor Xu
        3. HBASE-13985-v3.patch
          4 kB
          Victor Xu

          Activity

            People

            • Assignee:
              victorunique Victor Xu
              Reporter:
              victorunique Victor Xu
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: