Uploaded image for project: 'Apache Nemo'
  1. Apache Nemo
  2. NEMO-64

Fix map stage hang under DataSkewPolicy

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 0.1

    Description

      Hundreds of GB scale MapReduce application running with DataSkewPolicy suffers occasional map stage hang - one map stage TaskGroup doesn't gets finished forever.

      It seems that there might be a race condition in the process of handling map stage TaskGroup state from ON_HOLD to COMPLETE, but we need to find the root cause of it.

      Attachments

        Issue Links

          blocks

          Umbrella - An overarching type made of sub-tasks NEMO-60 IR-based dynamic optimization

          • Major - Major loss of function.
          • Resolved
          is blocked by

          Bug - A problem which impairs or prevents the functions of the product. NEMO-181 Fix DataSkewPolicy bug for multiple shuffles

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link GitHub Pull Request #99

          Activity

            People

              jeongyoon Jeongyoon Eo
              jeongyoon Jeongyoon Eo
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: