Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-24859

Optimize in-memory representation of mapreduce TableSplit objects

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha-1, 2.3.3, 1.7.0, 2.4.0, 2.2.7
    • Fix Version/s: 3.0.0-alpha-1, 1.7.0, 2.4.0, 2.2.7, 2.3.4
    • Component/s: mapreduce
    • Labels:
      None

      Description

      It has been observed that when the table has too many regions, MR jobs consume a lot of memory in the client. This is because we keep the region level information in memory and the memory heavy object is TableSplit because of the Scan object as a part of it.

      However, it looks like the TableInputFormat for single table doesn't need to store the scan object in the TableSplit because we do not use it and all the splits are expected to have the exact same scan object. In TableInputFormat we use the scan object directly from the MR conf.

        Attachments

        1. hbase-24859.png
          487 kB
          Rushabh Shah
        2. Screen Shot 2020-08-26 at 8.44.34 AM.png
          331 kB
          Sandeep Pal

          Issue Links

            Activity

              People

              • Assignee:
                sandeep.pal Sandeep Pal
                Reporter:
                sandeep.pal Sandeep Pal
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: