Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 1.2.0, 2.0.2-alpha
    • Component/s: native, performance
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      In HADOOP-7714, we determined that using fadvise inside of the MapReduce shuffle can yield very good performance improvements. But many parts of the shuffle are FileSystem-agnostic and thus operate on FSDataInputStreams and RawLocalFileSystems. This JIRA is to figure out how to allow RawLocalFileSystem to expose its FileDescriptor object without unnecessarily polluting the public APIs.

      1. HADOOP-7754_branch-1_rev2.patch
        5 kB
        Ahmed Radwan
      2. HADOOP-7754_branch-1_rev3.patch
        5 kB
        Ahmed Radwan
      3. HADOOP-7754_branch-1_rev4.patch
        6 kB
        Ahmed Radwan
      4. HADOOP-7754_trunk_rev2.patch
        6 kB
        Ahmed Radwan
      5. HADOOP-7754_trunk_rev2.patch
        6 kB
        Ahmed Radwan
      6. HADOOP-7754_trunk_rev3.patch
        6 kB
        Ahmed Radwan
      7. HADOOP-7754_trunk_rev4.patch
        7 kB
        Ahmed Radwan
      8. HADOOP-7754_trunk_rev4.patch
        7 kB
        Ahmed Radwan
      9. HADOOP-7754_trunk_rev4.patch
        7 kB
        Ahmed Radwan
      10. HADOOP-7754_trunk.patch
        5 kB
        Ahmed Radwan
      11. hadoop-7754-0.23.0-hasfd.txt
        4 kB
        Cristina L. Abad
      12. hasfd.txt
        4 kB
        Todd Lipcon
      13. test-patch-hadoop-7754.txt
        168 kB
        Ahmed Radwan

        Issue Links

          Activity

          Gavin made changes -
          Link This issue is depended upon by MAPREDUCE-4511 [ MAPREDUCE-4511 ]
          Gavin made changes -
          Link This issue blocks MAPREDUCE-4511 [ MAPREDUCE-4511 ]
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Alejandro Abdelnur made changes -
          Fix Version/s 1.2.0 [ 12321659 ]
          Alejandro Abdelnur made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Fix Version/s 2.2.0-alpha [ 12322473 ]
          Resolution Fixed [ 1 ]
          Ahmed Radwan made changes -
          Attachment HADOOP-7754_trunk_rev4.patch [ 12540984 ]
          Ahmed Radwan made changes -
          Attachment test-patch-hadoop-7754.txt [ 12540912 ]
          Ahmed Radwan made changes -
          Attachment HADOOP-7754_trunk_rev4.patch [ 12540799 ]
          Suresh Srinivas made changes -
          Component/s performance [ 12316502 ]
          Ahmed Radwan made changes -
          Attachment MAPREDUCE-4511_trunk_rev4.patch [ 12540731 ]
          Ahmed Radwan made changes -
          Attachment HADOOP-7754_trunk_rev4.patch [ 12540733 ]
          Ahmed Radwan made changes -
          Attachment MAPREDUCE-4511_trunk_rev4.patch [ 12540731 ]
          Ahmed Radwan made changes -
          Attachment HADOOP-7754_branch-1_rev4.patch [ 12540730 ]
          Ahmed Radwan made changes -
          Attachment HADOOP-7754_trunk_rev3.patch [ 12539932 ]
          Ahmed Radwan made changes -
          Attachment HADOOP-7754_branch-1_rev3.patch [ 12539931 ]
          Ahmed Radwan made changes -
          Attachment HADOOP-7754_trunk_rev2.patch [ 12539921 ]
          Ahmed Radwan made changes -
          Attachment HADOOP-7754_trunk_rev2.patch [ 12539739 ]
          Attachment HADOOP-7754_branch-1_rev2.patch [ 12539740 ]
          Aaron T. Myers made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Ahmed Radwan made changes -
          Attachment HADOOP-7754_trunk.patch [ 12539251 ]
          Ahmed Radwan made changes -
          Link This issue blocks MAPREDUCE-4511 [ MAPREDUCE-4511 ]
          Cristina L. Abad made changes -
          Attachment hadoop-7754-0.23.0-hasfd.txt [ 12532721 ]
          Todd Lipcon made changes -
          Attachment hasfd.txt [ 12499491 ]
          Todd Lipcon made changes -
          Affects Version/s 0.23.0 [ 12315569 ]
          Target Version/s 0.23.0 [ 12315569 ]
          Description In HADOOP-7714, we determined that using fadvise inside of the MapReduce shuffle can yield very good performance improvements. But many parts of the shuffle are FileSystem-agnostic and thus operate on FSDataInputStreams and RawLocalFileSystems. This JIRA is to figure out how to allow RawLocalFileSystem to expose its FileDescriptor object without unnecessarily polluting the public APIs.
          Todd Lipcon made changes -
          Field Original Value New Value
          Component/s native [ 12312070 ]
          Todd Lipcon created issue -

            People

            • Assignee:
              Todd Lipcon
              Reporter:
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              17 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development