[HDFS-14986] ReplicaCachingGetSpaceUsed throws ConcurrentModificationException

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.10.0
• Fix Version/s: 3.3.0, 3.1.4, 2.10.1
• Component/s: datanode, performance
• Hadoop Flags: Reviewed
• Assignee: Aiphago
• Reporter: Ryan Wu
• Created: 13/Nov/19 12:02
• Updated: 02/Apr/20 19:45
• Resolved: 28/Nov/19 02:56

Description

Running DU across lots of disks is very expensive . We applied the patch HDFS-14313 to get  used space from ReplicaInfo in memory.However, new du threads throw the exception

// 2019-11-08 18:07:13,858 ERROR [refreshUsed-/home/vipshop/hard_disk/7/dfs/dn/current/BP-1203969992-XXXX-1450855658517] org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.ReplicaCachingGetSpaceUsed: ReplicaCachingGetSpaceUsed refresh error
java.util.ConcurrentModificationException: Tree has been modified outside of iterator    
at org.apache.hadoop.hdfs.util.FoldedTreeSet$TreeSetIterator.checkForModification(FoldedTreeSet.java:311)    
at org.apache.hadoop.hdfs.util.FoldedTreeSet$TreeSetIterator.hasNext(FoldedTreeSet.java:256)    
at java.util.AbstractCollection.addAll(AbstractCollection.java:343)    
at java.util.HashSet.<init>(HashSet.java:120)    
at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.deepCopyReplica(FsDatasetImpl.java:1052)    
at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.ReplicaCachingGetSpaceUsed.refresh(ReplicaCachingGetSpaceUsed.java:73)    
at org.apache.hadoop.fs.CachingGetSpaceUsed$RefreshThread.run(CachingGetSpaceUsed.java:178)   
at java.lang.Thread.run(Thread.java:748)

Issue Links

• is caused by: HDFS-14313 Get hdfs used space from FsDatasetImpl#volumeMap#ReplicaInfo in memory instead of df/du