Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5791

Shuffle phase is slow in Windows - FadviseFileRegion::transferTo does not read disks efficiently

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 2.3.0, 3.0.0-alpha1
    • 2.4.0
    • client
    • None
    • Reviewed

    Description

      transferTo method in org.apache.hadoop.mapred.FadvisedFileRegion is using transferTo method from a FileChannel to transfer data from a disk to socket. This is performing slow in Windows, slower than in Linux. The reason is that transferTo method for the java.nio is issuing 32K IO requests all the time. In Windows, these 32K transfers are not optimal and we don't get the best performance form the underlying IO subsystem. In order to achieve better performance when reading from the drives, we need to read data in bigger chunks, 512K for example.

      Attachments

        1. MAPREDUCE-5791.patch
          16 kB
          Nikola Vujic
        2. MAPREDUCE-5791.patch
          15 kB
          Nikola Vujic
        3. MAPREDUCE-5791.patch
          14 kB
          Nikola Vujic

        Issue Links

          Activity

            People

              nikola.vujic Nikola Vujic
              nikola.vujic Nikola Vujic
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: