Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-20045

Make sure SparkHadoopMapReduceWriter is resilient to failures of writers and committers

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Incomplete
    • 2.2.0
    • None
    • Spark Core

    Description

      Following on from SPARK-20038: review SparkHadoopMapReduceWriter and ensure that it's failure handling code is itself resilient to follow on failures, especially in things like writer.close() and the abortTask/abortJob calls. That's to ensure as robust a cleanup as possible, and to stop the original exception getting lost.

      At a quick glance

      1. executeTask()'s catch logic should catch & log any failure in the writer.close()
      2. The Hadoop commit protocol's abort* operation's can throw IOEs. Again, they need to be caught and logged

      Should be testable with mocking, and worthwhile giving how important commit protocol resilience is.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              stevel@apache.org Steve Loughran
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: