Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-14339

HBase Bulk Load and super wide rows

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      This may not be a huge issues but it does come up. If the number of columns in a row are to many then KeyValueSortReducer will blow up with a out of memory exception, because it uses a TreeMap to sort the columns with in the memory of the reducer.

      A solution would be to add the column family and qualifier to the key so the shuffle would handle the sort.

      The partitioner would only partition on the rowKey but ordering would apply to the RowKey, Column Family, and Column Qualifier.

      Look at the Spark Bulk load as an example. HBASE-14150

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                ted.m Theodore michael Malaska
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: