HBase
  1. HBase
  2. HBASE-4552

multi-CF bulk load is not atomic across column families

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.92.0
    • Fix Version/s: 0.92.0
    • Component/s: regionserver
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change
    • Release Note:
      Hide
      HRegion.bulkLoadHFile() has been removed in 0.92 and TRUNK.
      HRegion.bulkLoadHFiles() is introduced for multi-family bulk load.
      Show
      HRegion.bulkLoadHFile() has been removed in 0.92 and TRUNK. HRegion.bulkLoadHFiles() is introduced for multi-family bulk load.

      Description

      Currently the bulk load API simply imports one HFile at a time. With multi-column-family support, this is inappropriate, since different CFs show up separately. Instead, the IPC endpoint should take a of CF -> HFiles, so we can online them all under a single region-wide lock.

      1. hbase-4552.consolidated.patch
        55 kB
        Jonathan Hsieh
      2. hbase-4552.consolidated.v2.patch
        55 kB
        Jonathan Hsieh
      3. hbase-4552.consolidated.v3.patch
        55 kB
        Ted Yu
      4. hbase-4552.consolidated.v4.patch
        53 kB
        Ted Yu

        Issue Links

          Activity

          Todd Lipcon created issue -
          Ted Yu made changes -
          Field Original Value New Value
          Comment [ The above optimization for reducing calls to region server can be done in a seperate JIRA.
          server.bulkLoadHFile() expects name of the region where HFile fits. Region name resolution needs to call conn.getRegionServerWithRetries(). ]
          Ted Yu made changes -
          Comment [ Toward the end of LoadIncrementalHFiles.tryLoad() we can utilize startEndKeys of the underlying table to group (possibly split) HFiles by their first row keys.
          Then the new bulkLoadHFile() method would be called by doBulkLoad(). ]
          Jonathan Hsieh made changes -
          Assignee Jonathan Hsieh [ jmhsieh ]
          Jonathan Hsieh made changes -
          Link This issue blocks HBASE-4652 [ HBASE-4652 ]
          Jonathan Hsieh made changes -
          Attachment hbase-4552.consolidated.patch [ 12501390 ]
          Jonathan Hsieh made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Jonathan Hsieh made changes -
          Attachment hbase-4552.consolidated.v2.patch [ 12501415 ]
          Ted Yu made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Ted Yu made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Ted Yu made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Ted Yu made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Ted Yu made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Ted Yu made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Ted Yu made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Ted Yu made changes -
          Resolution Fixed [ 1 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Ted Yu made changes -
          Comment [ Test failures in TRUNK is down to 1. ]
          Ted Yu made changes -
          Attachment hbase-4552.consolidated.v3.patch [ 12501459 ]
          Ted Yu made changes -
          Status Reopened [ 4 ] Patch Available [ 10002 ]
          Ted Yu made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Ted Yu made changes -
          Attachment hbase-4552.consolidated.v4.patch [ 12501481 ]
          Ted Yu made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Ted Yu made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Ted Yu made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Ted Yu made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Incompatible change [ 10342 ]
          Release Note HRegion.bulkLoadHFile() has been removed in 0.92 and TRUNK.
          HRegion.bulkLoadHFiles() is introduced for multi-family bulk load.
          Resolution Fixed [ 1 ]
          Jonathan Hsieh made changes -
          Link This issue is cloned as HBASE-4718 [ HBASE-4718 ]
          Gavin made changes -
          Link This issue blocks HBASE-4652 [ HBASE-4652 ]
          Gavin made changes -
          Link This issue is depended upon by HBASE-4652 [ HBASE-4652 ]

            People

            • Assignee:
              Jonathan Hsieh
              Reporter:
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development