Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Allow ReduceTask loading a third party plugin for shuffle (and merge) instead of the default shuffle.

      Description

      Support generic shuffle service as set of two plugins: ShuffleProvider & ShuffleConsumer.
      This will satisfy the following needs:

      1. Better shuffle and merge performance. For example: we are working on shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, or Infiniband) instead of using the current HTTP shuffle. Based on the fast RDMA shuffle, the plugin can also utilize a suitable merge approach during the intermediate merges. Hence, getting much better performance.
      2. Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden dependency of NodeManager with a specific version of mapreduce shuffle (currently targeted to 0.24.0).

      References:

      1. Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu from Auburn University with others, http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf
      2. I am attaching 2 documents with suggested Top Level Design for both plugins (currently, based on 1.0 branch)
      3. I am providing link for downloading UDA - Mellanox's open source plugin that implements generic shuffle service using RDMA and levitated merge. Note: At this phase, the code is in C++ through JNI and you should consider it as beta only. Still, it can serve anyone that wants to implement or contribute to levitated merge. (Please be advised that levitated merge is mostly suit in very fast networks) - http://www.mellanox.com/content/pages.php?pg=products_dyn&product_family=144&menu_section=69
      1. MAPREDUCE-4049--branch-1.patch
        56 kB
        Avner BenHanoch
      2. MAPREDUCE-4049--branch-1.patch
        54 kB
        Avner BenHanoch
      3. MAPREDUCE-4049--branch-1.patch
        27 kB
        Avner BenHanoch
      4. mapreduce-4049.patch
        25 kB
        Avner BenHanoch
      5. Hadoop Shuffle Plugin Design.rtf
        78 kB
        Avner BenHanoch
      6. HADOOP-1.x.y.patch
        20 kB
        Avner BenHanoch

        Issue Links

          Activity

          Alejandro Abdelnur made changes -
          Fix Version/s 1.3.0 [ 12324153 ]
          Avner BenHanoch made changes -
          Link This issue is related to YARN-802 [ YARN-802 ]
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Alejandro Abdelnur made changes -
          Release Note Allow ReduceTask loading a third party plugin for shuffle (and merge) instead of the default shuffle. A corresponding ShuffleProvider is anyhow allowed to run in the NM as an AuxiliaryService.
          Use new config option: mapreduce.job.reduce.shuffle.consumer.plugin.class - Name of the class whose instance will be used to send shuffle requests by reducetasks of this job. The class must be an instance of org.apache.hadoop.mapred.ShuffleConsumerPlugin.
          Allow ReduceTask loading a third party plugin for shuffle (and merge) instead of the default shuffle.
          Alejandro Abdelnur made changes -
          Link This issue is related to MAPREDUCE-4977 [ MAPREDUCE-4977 ]
          Avner BenHanoch made changes -
          Attachment MAPREDUCE-4049--branch-1.patch [ 12567973 ]
          Avner BenHanoch made changes -
          Attachment MAPREDUCE-4049--branch-1.patch [ 12566960 ]
          Avner BenHanoch made changes -
          Release Note Support Shuffle Consumer plugins from 3rd parties. Allow ReduceTask loading a third party plugin for shuffle (and merge) instead of the default shuffle. A corresponding ShuffleProvider is anyhow allowed to run in the NM as an AuxiliaryService.
          Use new config option: mapreduce.job.reduce.shuffle.consumer.plugin.class - Name of the class whose instance will be used to send shuffle requests by reducetasks of this job. The class must be an instance of org.apache.hadoop.mapred.ShuffleConsumerPlugin.
          Alejandro Abdelnur made changes -
          Fix Version/s 2.0.3-alpha [ 12323275 ]
          Fix Version/s 3.0.0 [ 12320355 ]
          Avner BenHanoch made changes -
          Attachment MAPREDUCE-4049--branch-1.patch [ 12565121 ]
          Harsh J made changes -
          Comment [ I’ll be on vacation between Jan 6 to 13 (returning on Monday the 14th)
          Redirecting issues:
          · VMA -> Olga Shern olgas@mellanox.com<mailto:olgas@mellanox.com>
          · UDA -> Avner Ben Hanoch avnerb@mellanox.com<mailto:avnerb@mellanox.com>

          Regards,

          Alex Rosenbaum
          Director R&D Application Acceleration
          Mellanox Technologies
          13 Zarhin st, Raanana, Israel
          +972 (74) 712-9215

          Follow us on Twitter<http://twitter.com/mellanoxtech> and Facebook<http://www.facebook.com/pages/Mellanox-Technologies/223164879116>
          ]
          Avner BenHanoch made changes -
          Attachment mapreduce-4049.patch [ 12556383 ]
          Avner BenHanoch made changes -
          Attachment mapreduce-4049.patch [ 12552079 ]
          Avner BenHanoch made changes -
          Attachment mapreduce-4049.patch [ 12552017 ]
          Avner BenHanoch made changes -
          Attachment mapreduce-4049.patch [ 12542917 ]
          Avner BenHanoch made changes -
          Attachment mapreduce-4049.patch [ 12542778 ]
          Alejandro Abdelnur made changes -
          Status Reopened [ 4 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Alejandro Abdelnur made changes -
          Resolution Fixed [ 1 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Alejandro Abdelnur made changes -
          Parent MAPREDUCE-2454 [ 12505228 ]
          Issue Type Improvement [ 4 ] Sub-task [ 7 ]
          Arun C Murthy made changes -
          Parent MAPREDUCE-2454 [ 12505228 ]
          Issue Type Sub-task [ 7 ] Improvement [ 4 ]
          Arun C Murthy made changes -
          Fix Version/s 3.0.0 [ 12320355 ]
          Fix Version/s MR-2454 [ 12323590 ]
          Alejandro Abdelnur made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Fix Version/s MR-2454 [ 12323590 ]
          Fix Version/s trunk [ 12320360 ]
          Resolution Fixed [ 1 ]
          Avner BenHanoch made changes -
          Attachment mapreduce-4049.patch [ 12559764 ]
          Avner BenHanoch made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Avner BenHanoch made changes -
          Attachment mapreduce-4049.patch [ 12556383 ]
          Alejandro Abdelnur made changes -
          Parent MAPREDUCE-2454 [ 12505228 ]
          Issue Type Improvement [ 4 ] Sub-task [ 7 ]
          Alejandro Abdelnur made changes -
          Assignee Avner BenHanoch [ avnerb ]
          Arun C Murthy made changes -
          Parent MAPREDUCE-2454 [ 12505228 ]
          Issue Type Sub-task [ 7 ] Improvement [ 4 ]
          Avner BenHanoch made changes -
          Description Support generic shuffle service as set of two plugins: ShuffleProvider & ShuffleConsumer.
          This will satisfy the following needs:
          # Better shuffle and merge performance. For example: we are working on shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, or Infiniband) instead of using the current HTTP shuffle. Based on the fast RDMA shuffle, the plugin can also utilize a suitable merge approach during the intermediate merges. Hence, getting much better performance.
          # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden dependency of NodeManager with a specific version of mapreduce shuffle (currently targeted to 0.24.0).

          References:
          # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu from Auburn University with others, [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf]
          # I am attaching 2 documents with suggested Top Level Design for both plugins (currently, based on 1.0 branch)
          Support generic shuffle service as set of two plugins: ShuffleProvider & ShuffleConsumer.
          This will satisfy the following needs:
          # Better shuffle and merge performance. For example: we are working on shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, or Infiniband) instead of using the current HTTP shuffle. Based on the fast RDMA shuffle, the plugin can also utilize a suitable merge approach during the intermediate merges. Hence, getting much better performance.
          # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden dependency of NodeManager with a specific version of mapreduce shuffle (currently targeted to 0.24.0).

          References:
          # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu from Auburn University with others, [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf]
          # I am attaching 2 documents with suggested Top Level Design for both plugins (currently, based on 1.0 branch)
          # I am providing link for downloading UDA - Mellanox's open source plugin that implements generic shuffle service using RDMA and levitated merge. Note: At this phase, the code is in C++ through JNI and you should consider it as beta only. Still, it can serve anyone that wants to implement or contribute to levitated merge. (Please be advised that levitated merge is mostly suit in very fast networks) - [http://www.mellanox.com/content/pages.php?pg=products_dyn&product_family=144&menu_section=69]
          Alejandro Abdelnur made changes -
          Parent MAPREDUCE-2454 [ 12505228 ]
          Issue Type Improvement [ 4 ] Sub-task [ 7 ]
          Alejandro Abdelnur made changes -
          Link This issue is related to MAPREDUCE-4812 [ MAPREDUCE-4812 ]
          Arun C Murthy made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Avner BenHanoch made changes -
          Attachment mapreduce-4049.patch [ 12552079 ]
          Avner BenHanoch made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Fix Version/s trunk [ 12320360 ]
          Avner BenHanoch made changes -
          Attachment mapreduce-4049.patch [ 12552017 ]
          Avner BenHanoch made changes -
          Attachment Hadoop Shuffle Plugin Design.rtf [ 12550258 ]
          Avner BenHanoch made changes -
          Attachment mapred-site.xml [ 12521520 ]
          Avner BenHanoch made changes -
          Attachment Hadoop Shuffle Provider Plugin TLD.rtf [ 12519444 ]
          Avner BenHanoch made changes -
          Attachment Hadoop Shuffle Consumer Plugin TLD.rtf [ 12519445 ]
          Arun C Murthy made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Avner BenHanoch made changes -
          Attachment mapreduce-4049.patch [ 12542917 ]
          Avner BenHanoch made changes -
          Attachment mapreduce-4049.patch [ 12542769 ]
          Avner BenHanoch made changes -
          Attachment mapreduce-4049.patch [ 12542778 ]
          Avner BenHanoch made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Release Note Support plugin-able Shuffle Provider(s) and a matching plugin-able Shuffle Consumer(s). Support Shuffle Consumer plugins from 3rd parties.
          Target Version/s 1.1.0, 2.0.0-alpha, 3.0.0 [ 12317960, 12320354, 12320355 ] 3.0.0 [ 12320355 ]
          Avner BenHanoch made changes -
          Attachment mapreduce-4049.patch [ 12542769 ]
          Avner BenHanoch made changes -
          Attachment HADOOP-1.x.y.patch [ 12537213 ]
          Avner BenHanoch made changes -
          Attachment HADOOP-1.x.y-review-oriented.patch [ 12533652 ]
          Avner BenHanoch made changes -
          Attachment HADOOP-1.1.patch [ 12530039 ]
          Avner BenHanoch made changes -
          Attachment HADOOP-1.0.x.patch [ 12522757 ]
          Avner BenHanoch made changes -
          Attachment HADOOP-1.0.2.patch [ 12521533 ]
          Avner BenHanoch made changes -
          Attachment HADOOP-1.x.y.patch [ 12537729 ]
          Avner BenHanoch made changes -
          Attachment HADOOP-1.x.y.patch [ 12537213 ]
          Avner BenHanoch made changes -
          Attachment HADOOP-1.x.y-review-oriented.patch [ 12533652 ]
          Avner BenHanoch made changes -
          Attachment mapred.diff [ 12521519 ]
          Avner BenHanoch made changes -
          Attachment MAPREDUCE-4049-branch-1.0.2.patch [ 12521536 ]
          Avner BenHanoch made changes -
          Attachment src.tgz [ 12521521 ]
          Avner BenHanoch made changes -
          Attachment test.diff [ 12521522 ]
          Avner BenHanoch made changes -
          Attachment HADOOP-1.0.x.patch [ 12521534 ]
          Avner BenHanoch made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Target Version/s 1.1.0, 2.0.0-alpha, 3.0.0 [ 12317960, 12320354, 12320355 ] 2.0.0-alpha, 1.1.0, 3.0.0 [ 12320354, 12317960, 12320355 ]
          Avner BenHanoch made changes -
          Attachment HADOOP-1.1.patch [ 12530039 ]
          Avner BenHanoch made changes -
          Target Version/s 2.0.0, 1.1.0 [ 12320354, 12317960 ] 1.1.0, 2.0.0, 3.0.0 [ 12317960, 12320354, 12320355 ]
          Matt Foley made changes -
          Target Version/s 1.0.3 [ 12320250 ] 1.1.0, 2.0.0 [ 12317960, 12320354 ]
          Avner BenHanoch made changes -
          Affects Version/s 1.1.0 [ 12317960 ]
          Affects Version/s 2.0.0 [ 12320354 ]
          Affects Version/s 3.0.0 [ 12320355 ]
          Avner BenHanoch made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Affects Version/s 1.0.3 [ 12320250 ]
          Affects Version/s 0.23.1 [ 12318883 ]
          Affects Version/s 1.0.1 [ 12319503 ]
          Target Version/s 1.0.3, 0.23.3, 0.23.2, 1.1.0, 0.24.0 [ 12320250, 12320060, 12319851, 12317960, 12317654 ] 1.0.3 [ 12320250 ]
          Avner BenHanoch made changes -
          Attachment HADOOP-1.0.x.patch [ 12522757 ]
          Avner BenHanoch made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Target Version/s 1.0.3, 0.23.3, 0.23.2, 1.1.0, 0.24.0 [ 12320250, 12320060, 12319851, 12317960, 12317654 ] 0.24.0, 1.1.0, 0.23.2, 0.23.3, 1.0.3 [ 12317654, 12317960, 12319851, 12320060, 12320250 ]
          Jeff Hammerbacher made changes -
          Link This issue is related to MAPREDUCE-3060 [ MAPREDUCE-3060 ]
          Jeff Hammerbacher made changes -
          Link This issue is related to MAPREDUCE-2454 [ MAPREDUCE-2454 ]
          Matt Foley made changes -
          Fix Version/s 1.0.2 [ 12320047 ]
          Target Version/s 1.0.3, 0.23.3, 1.0.2, 0.23.2, 1.1.0, 0.24.0 [ 12320250, 12320060, 12320047, 12319851, 12317960, 12317654 ] 0.24.0, 1.1.0, 0.23.2, 0.23.3, 1.0.3 [ 12317654, 12317960, 12319851, 12320060, 12320250 ]
          Avner BenHanoch made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Target Version/s 1.0.3, 0.23.3, 1.0.2, 0.23.2, 1.1.0, 0.24.0 [ 12320250, 12320060, 12320047, 12319851, 12317960, 12317654 ] 0.24.0, 1.1.0, 0.23.2, 1.0.2, 0.23.3, 1.0.3 [ 12317654, 12317960, 12319851, 12320047, 12320060, 12320250 ]
          Avner BenHanoch made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Target Version/s 1.0.3, 0.23.3, 1.0.2, 0.23.2, 1.1.0, 0.24.0 [ 12320250, 12320060, 12320047, 12319851, 12317960, 12317654 ] 0.24.0, 1.1.0, 0.23.2, 1.0.2, 0.23.3, 1.0.3 [ 12317654, 12317960, 12319851, 12320047, 12320060, 12320250 ]
          Avner BenHanoch made changes -
          Attachment MAPREDUCE-4049-branch-1.0.2.patch [ 12521536 ]
          Avner BenHanoch made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Target Version/s 1.0.3, 0.23.3, 1.0.2, 0.23.2, 1.1.0, 0.24.0 [ 12320250, 12320060, 12320047, 12319851, 12317960, 12317654 ] 0.24.0, 1.1.0, 0.23.2, 1.0.2, 0.23.3, 1.0.3 [ 12317654, 12317960, 12319851, 12320047, 12320060, 12320250 ]
          Avner BenHanoch made changes -
          Attachment HADOOP-1.0.x.patch [ 12521534 ]
          Avner BenHanoch made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Target Version/s 1.0.3, 0.23.3, 1.0.2, 0.23.2, 1.1.0, 0.24.0 [ 12320250, 12320060, 12320047, 12319851, 12317960, 12317654 ] 0.24.0, 1.1.0, 0.23.2, 1.0.2, 0.23.3, 1.0.3 [ 12317654, 12317960, 12319851, 12320047, 12320060, 12320250 ]
          Avner BenHanoch made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Target Version/s 1.0.3, 0.23.3, 1.0.2, 0.23.2, 1.1.0, 0.24.0 [ 12320250, 12320060, 12320047, 12319851, 12317960, 12317654 ] 0.24.0, 1.1.0, 0.23.2, 1.0.2, 0.23.3, 1.0.3 [ 12317654, 12317960, 12319851, 12320047, 12320060, 12320250 ]
          Fix Version/s 1.0.2 [ 12320047 ]
          Fix Version/s 1.0.0 [ 12318240 ]
          Avner BenHanoch made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Target Version/s 1.0.3, 0.23.3, 1.0.2, 0.23.2, 1.1.0, 0.24.0 [ 12320250, 12320060, 12320047, 12319851, 12317960, 12317654 ] 0.24.0, 1.1.0, 0.23.2, 1.0.2, 0.23.3, 1.0.3 [ 12317654, 12317960, 12319851, 12320047, 12320060, 12320250 ]
          Avner BenHanoch made changes -
          Attachment HADOOP-1.0.2.patch [ 12521533 ]
          Avner BenHanoch made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Target Version/s 1.0.3, 0.23.3, 1.0.2, 0.23.2, 1.1.0, 0.24.0 [ 12320250, 12320060, 12320047, 12319851, 12317960, 12317654 ] 0.24.0, 1.1.0, 0.23.2, 1.0.2, 0.23.3, 1.0.3 [ 12317654, 12317960, 12319851, 12320047, 12320060, 12320250 ]
          Fix Version/s 1.0.0 [ 12318240 ]
          Avner BenHanoch made changes -
          Attachment mapred.diff [ 12521519 ]
          Attachment mapred-site.xml [ 12521520 ]
          Attachment src.tgz [ 12521521 ]
          Attachment test.diff [ 12521522 ]
          Avner BenHanoch made changes -
          Target Version/s 1.0.3, 0.23.3, 1.0.2, 0.23.2, 1.1.0, 0.24.0 [ 12320250, 12320060, 12320047, 12319851, 12317960, 12317654 ] 0.24.0, 1.1.0, 0.23.2, 1.0.2, 0.23.3, 1.0.3 [ 12317654, 12317960, 12319851, 12320047, 12320060, 12320250 ]
          Description Support generic shuffle service as set of two plugins: ShuffleProvider & ShuffleConsumer.
          This will satisfy the following needs:
          # Better shuffle and merge performance. For example: we are working on shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, or Infiniband) instead of using the current HTTP shuffle. Based on the fast RDMA shuffle, the plugin can also utilize a suitable merge approach during the intermediate merges. Hence, getting much better performance.
          # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden dependency of NodeManager with a specific version of mapreduce shuffle (currently targeted to 0.24.0).

          References:
          # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu from Auburn University with others, [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf]
          # I will soon attach document with suggested API for the plugin
          Support generic shuffle service as set of two plugins: ShuffleProvider & ShuffleConsumer.
          This will satisfy the following needs:
          # Better shuffle and merge performance. For example: we are working on shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, or Infiniband) instead of using the current HTTP shuffle. Based on the fast RDMA shuffle, the plugin can also utilize a suitable merge approach during the intermediate merges. Hence, getting much better performance.
          # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden dependency of NodeManager with a specific version of mapreduce shuffle (currently targeted to 0.24.0).

          References:
          # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu from Auburn University with others, [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf]
          # I am attaching 2 documents with suggested Top Level Design for both plugins (currently, based on 1.0 branch)
          Avner BenHanoch made changes -
          Attachment Hadoop Shuffle Provider Plugin TLD.rtf [ 12519444 ]
          Attachment Hadoop Shuffle Consumer Plugin TLD.rtf [ 12519445 ]
          Avner BenHanoch made changes -
          Target Version/s 1.0.3, 0.23.3, 1.0.2, 0.23.2, 1.1.0, 0.24.0 [ 12320250, 12320060, 12320047, 12319851, 12317960, 12317654 ] 0.24.0, 1.1.0, 0.23.2, 1.0.2, 0.23.3, 1.0.3 [ 12317654, 12317960, 12319851, 12320047, 12320060, 12320250 ]
          Description Support generic shuffle service as set of two plugins: ShuffleProvider & ShuffleConsumer.
          This will satisfy the following needs:
          # Better shuffle and merge performance. For example: we (Mellanox with Prof. Weikuan Yu from Auburn University) are working on shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, or Infiniband) instead of using the current HTTP shuffle. Based on the fast RDMA shuffle, the plugin can also utilize a suitable merge approach during the intermediate merges. Hence, getting much better performance.
          # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden dependency of NodeManager with a specific version of mapreduce shuffle (currently targeted to 0.24.0).

          References:
          # Hadoop Acceleration through Network Levitated Merging, [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf]
          # I will soon attach document with suggested API for the plugin
          Support generic shuffle service as set of two plugins: ShuffleProvider & ShuffleConsumer.
          This will satisfy the following needs:
          # Better shuffle and merge performance. For example: we are working on shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, or Infiniband) instead of using the current HTTP shuffle. Based on the fast RDMA shuffle, the plugin can also utilize a suitable merge approach during the intermediate merges. Hence, getting much better performance.
          # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden dependency of NodeManager with a specific version of mapreduce shuffle (currently targeted to 0.24.0).

          References:
          # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu from Auburn University with others, [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf]
          # I will soon attach document with suggested API for the plugin
          Avner BenHanoch made changes -
          Field Original Value New Value
          Target Version/s 1.0.3, 0.23.3, 1.0.2, 0.23.2, 1.1.0, 0.24.0 [ 12320250, 12320060, 12320047, 12319851, 12317960, 12317654 ] 0.24.0, 1.1.0, 0.23.2, 1.0.2, 0.23.3, 1.0.3 [ 12317654, 12317960, 12319851, 12320047, 12320060, 12320250 ]
          Description Support generic shuffle service as set of two plugins: ShuffleProvider & ShuffleConsumer.
          This will satisfy the following needs:
          # Better shuffle and merge performance. For example: we (Mellanox with Prof. Weikuan Yu from Auburn University) are working on shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, or Infiniband) instead of using the current HTTP shuffle. Based on the fast RDMA shuffle, the plugin can also utilize a suitable merge approach during the intermediate merges. Hence, getting much better performance.
          # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden dependency of NodeManager with a specific version of mapreduce shuffle (currently targeted to 0.24.0).

          Reference:
          Hadoop Acceleration through Network Levitated Merging, [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf]
          Support generic shuffle service as set of two plugins: ShuffleProvider & ShuffleConsumer.
          This will satisfy the following needs:
          # Better shuffle and merge performance. For example: we (Mellanox with Prof. Weikuan Yu from Auburn University) are working on shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, or Infiniband) instead of using the current HTTP shuffle. Based on the fast RDMA shuffle, the plugin can also utilize a suitable merge approach during the intermediate merges. Hence, getting much better performance.
          # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden dependency of NodeManager with a specific version of mapreduce shuffle (currently targeted to 0.24.0).

          References:
          # Hadoop Acceleration through Network Levitated Merging, [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf]
          # I will soon attach document with suggested API for the plugin
          Avner BenHanoch created issue -

            People

            • Assignee:
              Avner BenHanoch
              Reporter:
              Avner BenHanoch
            • Votes:
              9 Vote for this issue
              Watchers:
              52 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development