Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1761

FairScheduler should allow separate configuration of node and rack locality wait time

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.22.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      It would be nice that we can separately assign rack locality wait time.
      In our use case, we would set node locality wait to zero and wait only rack locality.

      I propose that we add two parameters
      mapred.fairscheduler.locality.delay.nodetorack
      mapred.fairscheduler.locality.delay.racktoany
      This allows specifying the wait time on each stage.

      And we can use
      mapred.fairscheduler.locality.delay
      as the default value of the above fields so that this is backward compatible.

      Thoughts?

      1. MAPREDUCE-1761.txt
        6 kB
        Scott Chen
      2. MAPREDUCE-1761-v1.1.txt
        6 kB
        Scott Chen
      3. MAPREDUCE-1761-v1.2.txt
        6 kB
        Scott Chen

        Activity

        Scott Chen created issue -
        Scott Chen made changes -
        Field Original Value New Value
        Attachment MAPREDUCE-1761.txt [ 12444233 ]
        Scott Chen made changes -
        Attachment MAPREDUCE-1761-v1.1.txt [ 12444243 ]
        Scott Chen made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Scott Chen made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Scott Chen made changes -
        Attachment MAPREDUCE-1761-v1.2.txt [ 12444256 ]
        Scott Chen made changes -
        Attachment MAPREDUCE-1761-v1.2.txt [ 12444256 ]
        Scott Chen made changes -
        Attachment MAPREDUCE-1761-v1.2.txt [ 12444258 ]
        Scott Chen made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        dhruba borthakur made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags [Reviewed]
        Resolution Fixed [ 1 ]
        Konstantin Shvachko made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Scott Chen
            Reporter:
            Scott Chen
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development