Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-3912

Fetchers should be more robust to corrupted inputs

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.9.2, 0.10.0
    • None
    • None

    Description

      I recently saw a case where a bad node in the cluster produced corrupted shuffle data that caused the codec to throw IllegalArgumentException when trying to fetch. Fetchers currently only handle IOException and InternalError, and any other type of exception will cause the entire task to be torn down. We should consider catching Exception like MapReduce does to be more robust in light of other types of errors coming from the codec and allow retries to occur.

      Attachments

        1. TEZ-3912.001.patch
          3 kB
          Kuhu Shukla
        2. TEZ-3912.002.patch
          5 kB
          Kuhu Shukla

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. TEZ-3196 java.lang.InternalError from decompression codec is fatal to a task during shuffle

          • Major - Major loss of function.
          • Closed

          Activity

            People

              kshukla Kuhu Shukla
              jlowe Jason Darrell Lowe
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: