Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-4055

Client region location caches redundant HTableDescriptor's

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Duplicate
    • 0.90.3
    • 0.92.0
    • None
    • None

    Description

      While examining the heap of a map task in a MapReduce job that writes directly to HBase, I noticed that the HRegionLocation instances were taking up 90 MB (out of a 700 MB heap for each map task) to cache the locations for 15K regions. As the number of regions in the cluster continues to grow, this continues to grow as well.

      Of that, it appears that about 80 MB were going to 15K HTableDescriptor instances. There are only 5 tables that it's writing to, so it seems to be wasting a great deal of memory with a separate copy of the table descriptor for each region.

      Attachments

        Issue Links

          duplicates

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-451 Remove HTableDescriptor from HRegionInfo

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Activity

            People

              Unassigned Unassigned
              davelatham Dave Latham
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: