[MAPREDUCE-5368] Save memory by set capacity, load factor and concurrency level for ConcurrentHashMap in TaskInProgress

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.2.0
• Fix Version/s: 1.2.1
• Component/s: mrv1
• Assignee: yunjiong zhao
• Reporter: yunjiong zhao
• Created: 02/Jul/13 07:08
• Updated: 04/Aug/13 08:23
• Resolved: 05/Jul/13 03:24

Description

Below is histo from our JobTracker:

num #instances #bytes class name
----------------------------------------------
1: 136048824 11347237456 [C
2: 124156992 5959535616 java.util.concurrent.locks.ReentrantLock$NonfairSync
3: 124156973 5959534704 java.util.concurrent.ConcurrentHashMap$Segment
4: 135887753 5435510120 java.lang.String
5: 124213692 3975044400 [Ljava.util.concurrent.ConcurrentHashMap$HashEntry;
6: 63777311 3061310928 java.util.HashMap$Entry
7: 35038252 2803060160 java.util.TreeMap
8: 16921110 2712480072 [Ljava.util.HashMap$Entry;
9: 4803617 2420449192 [Ljava.lang.Object;
10: 50392816 2015712640 org.apache.hadoop.mapred.Counters$Counter
11: 7775438 1181866576 [Ljava.util.concurrent.ConcurrentHashMap$Segment;
12: 3882847 1118259936 org.apache.hadoop.mapred.TaskInProgress

ConcurrentHashMap takes more than 14G(5959535616 + 5959534704 + 3975044400).
The trouble maker are below codes in TaskInProgress.java:
Map<TaskAttemptID, Locality> taskLocality =
new ConcurrentHashMap<TaskAttemptID, Locality>();
Map<TaskAttemptID, Avataar> taskAvataar =
new ConcurrentHashMap<TaskAttemptID, Avataar>();