Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3721

Race in shuffle can cause it to hang

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Blocker
    • Resolution: Fixed
    • 0.23.0
    • 0.23.1
    • mrv2
    • None

    Description

      If all current {{Fetcher}}s complete while an in-memory merge is in progress - shuffle could hang.
      Specifically - if the memory freed by an in-memory merge does not bring MergeManager.usedMemory below MergeManager.memoryLimit and all current Fetchers complete before the in-memory merge completes, another in-memory merge will not be triggered - and shuffle will hang. (All new fetchers are asked to WAIT).

      Attachments

        1. MR-3721.txt
          4 kB
          Siddharth Seth

        Issue Links

          blocks

          Sub-task - The sub-task of the issue MAPREDUCE-3719 Make gridmix performance on YARN+MR to match or exceed that on 1.0

          • Major - Major loss of function.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-4842 Shuffle race can hang reducer

          • Blocker - Blocks development and/or testing work, production could not run
          • Closed

          Activity

            People

              sseth Siddharth Seth
              sseth Siddharth Seth
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: