[HDFS-14541] When evictableMmapped or evictable size is zero, do not throw NoSuchElementException

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.10.0, 3.0.4, 3.3.0, 3.2.1, 2.9.3, 3.1.3
• Component/s: hdfs-client, performance
• Hadoop Flags: Reviewed
• Assignee: Lisheng Sun
• Reporter: Zheng Hu
• Created: 04/Jun/19 09:48
• Updated: 11/Feb/20 08:06
• Resolved: 25/Jun/19 00:50

Description

Our XiaoMi HBase team are evaluating the performence improvement of HBASE-21879, and we have few CPU flame graph & heap flame graph by using async-profiler, and find that there're some performence issues in DFSClient .

See the attached two flame graphs, we can conclude that the try catch block in ShortCircuitCache#trimEvictionMaps has some serious perf problem , we should remove the try catch from DFSClient.

  /**
   * Trim the eviction lists.
   */
  private void trimEvictionMaps() {
    long now = Time.monotonicNow();
    demoteOldEvictableMmaped(now);

    while (true) {
      long evictableSize = evictable.size();
      long evictableMmappedSize = evictableMmapped.size();
      if (evictableSize + evictableMmappedSize <= maxTotalSize) {
        return;
      }
      ShortCircuitReplica replica;
      try {
        if (evictableSize == 0) {
          replica = (ShortCircuitReplica)evictableMmapped.get(evictableMmapped
              .firstKey());
        } else {
          replica = (ShortCircuitReplica)evictable.get(evictable.firstKey());
        }
      } catch (NoSuchElementException e) {
        break;
      }
      if (LOG.isTraceEnabled()) {
        LOG.trace(this + ": trimEvictionMaps is purging " + replica +
            StringUtils.getStackTrace(Thread.currentThread()));
      }
      purge(replica);
    }
  }

Our Xiaomi HDFS Team member leosun08 will prepare patch for this issue.

Issue Links

• is related to: HDFS-15161 When evictableMmapped or evictable size is zero, do not throw NoSuchElementException in ShortCircuitCache#close()
• relates to: HBASE-22387 Evaluate the get/scan performance after reading HFile block into offheap directly
• relates to: HBASE-21879 Read HFile's block to ByteBuffer directly instead of to byte for reducing young gc purpose
• links to: GitHub Pull Request #977