Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-16032

Possible memory leak in StoreScanner

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.2.1, 1.1.5, 0.98.20
    • 1.3.0, 1.2.2, 1.1.6, 0.98.21, 2.0.0
    • None
    • None
    • Reviewed

    Description

      We observed frequent fullGC of RS in our production environment, and after analyzing the heapdump, we found large memory occupancy by HStore#changedReaderObservers, the map is surprisingly containing 7500w objects...

      After some debugging, I located some possible memory leak in StoreScanner constructor:

        public StoreScanner(Store store, ScanInfo scanInfo, Scan scan, final NavigableSet<byte[]> columns,
            long readPt)
        throws IOException {
          this(store, scan, scanInfo, columns, readPt, scan.getCacheBlocks());
          if (columns != null && scan.isRaw()) {
            throw new DoNotRetryIOException("Cannot specify any column for a raw scan");
          }
          matcher = new ScanQueryMatcher(scan, scanInfo, columns,
              ScanType.USER_SCAN, Long.MAX_VALUE, HConstants.LATEST_TIMESTAMP,
              oldestUnexpiredTS, now, store.getCoprocessorHost());
      
          this.store.addChangedReaderObserver(this);
      
          // Pass columns to try to filter out unnecessary StoreFiles.
          List<KeyValueScanner> scanners = getScannersNoCompaction();
          ...
          seekScanners(scanners, matcher.getStartKey(), explicitColumnQuery
              && lazySeekEnabledGlobally, parallelSeekEnabled);
          ...
          resetKVHeap(scanners, store.getComparator());
        }
      

      If there's any Exception thrown after this.store.addChangedReaderObserver(this), the returned scanner might be null and there's no chance to remove the scanner from changedReaderObservers, like in HRegion#get

          RegionScanner scanner = null;
          try {
            scanner = getScanner(scan);
            scanner.next(results);
          } finally {
            if (scanner != null)
              scanner.close();
          }
      

      What's more, all exception thrown in the HRegion#getScanner path will cause scanner==null then memory leak, so we also need to handle this part.

      Attachments

        1. HBASE-16032_v2.patch
          4 kB
          Yu Li
        2. HBASE-16032_v3.patch
          4 kB
          Yu Li
        3. HBASE-16032_v4.patch
          5 kB
          Yu Li
        4. HBASE-16032.patch
          2 kB
          Yu Li

        Issue Links

          Activity

            People

              liyu Yu Li
              liyu Yu Li
              Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: