Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5791

Shuffle phase is slow in Windows - FadviseFileRegion::transferTo does not read disks efficiently

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.3.0, 3.0.0-alpha1
    • Fix Version/s: 2.4.0
    • Component/s: client
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      transferTo method in org.apache.hadoop.mapred.FadvisedFileRegion is using transferTo method from a FileChannel to transfer data from a disk to socket. This is performing slow in Windows, slower than in Linux. The reason is that transferTo method for the java.nio is issuing 32K IO requests all the time. In Windows, these 32K transfers are not optimal and we don't get the best performance form the underlying IO subsystem. In order to achieve better performance when reading from the drives, we need to read data in bigger chunks, 512K for example.

        Attachments

        1. MAPREDUCE-5791.patch
          14 kB
          Nikola Vujic
        2. MAPREDUCE-5791.patch
          15 kB
          Nikola Vujic
        3. MAPREDUCE-5791.patch
          16 kB
          Nikola Vujic

          Issue Links

            Activity

              People

              • Assignee:
                nikola.vujic Nikola Vujic
                Reporter:
                nikola.vujic Nikola Vujic
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: