[HDFS-16284] Unboxing long value in old version's block report cause full GC

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Labels: pull-request-available
• Assignee: Yuxuan Wang
• Reporter: Yuxuan Wang
• Created: 26/Oct/21 06:11
• Updated: 25/Dec/21 10:54

Description

When upgrade cluster from 2.6 to 3.1, namenode fall into full GC trouble.

NN run 3.1 version and DN run 2.6 version, block report type is longs not PB introduced by https://issues.apache.org/jira/browse/HDFS-7435 . So it's run into LongDecoder.

We find change org.apache.hadoop.hdfs.protocol.BlockListAsLongs.LongsDecoder#values type from List<Long> to long[] can fix the issue, but I can't for sure that the Long unboxing cause full GC problem.

Issue Links

• causes: HDFS-7435 PB encoding of block reports is very inefficient
• is related to: HDFS-11096 Support rolling upgrade between 2.x and 3.x
• links to: GitHub Pull Request #3586