[MAPREDUCE-5791] Shuffle phase is slow in Windows - FadviseFileRegion::transferTo does not read disks efficiently

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.0, 3.0.0-alpha1
• Fix Version/s: 2.4.0
• Component/s: client
• Target Version/s: 2.4.0
• Hadoop Flags: Reviewed
• Assignee: Nikola Vujic
• Reporter: Nikola Vujic
• Created: 11/Mar/14 14:07
• Updated: 02/Aug/17 21:33
• Resolved: 24/Mar/14 19:10

Description

transferTo method in org.apache.hadoop.mapred.FadvisedFileRegion is using transferTo method from a FileChannel to transfer data from a disk to socket. This is performing slow in Windows, slower than in Linux. The reason is that transferTo method for the java.nio is issuing 32K IO requests all the time. In Windows, these 32K transfers are not optimal and we don't get the best performance form the underlying IO subsystem. In order to achieve better performance when reading from the drives, we need to read data in bigger chunks, 512K for example.

Issue Links

• is related to: MAPREDUCE-6923 Optimize MapReduce Shuffle I/O for small partitions