Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-3990

The number of shuffle penalties for a host/inputAttemptIdentifier should be capped

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.9.1, 0.10.0
    • 0.9.2, 0.10.0
    • None
    • None

    Description

      In a scenario where the same mapId fetches fail, the penalty code allows adding the same Host/InputAttemptIdentifier over and over with revised penalty time that grows exponentially. It should at some point drop the retrying and report failure to the AM asap to allow the job to rectify the upstream output.

      Attachments

        1. TEZ-3990.006.patch
          8 kB
          Kuhu Shukla
        2. TEZ-3990.005.patch
          8 kB
          Kuhu Shukla
        3. TEZ-3990.004.patch
          7 kB
          Kuhu Shukla
        4. TEZ-3990.003.patch
          7 kB
          Kuhu Shukla
        5. TEZ-3990.002.patch
          6 kB
          Kuhu Shukla
        6. TEZ-3990.001.patch
          4 kB
          Kuhu Shukla

        Activity

          People

            kshukla Kuhu Shukla
            kshukla Kuhu Shukla
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: