Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-13793

PipedRDD doesn't propagate exceptions while reading parent RDDd

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.6.0
    • Fix Version/s: 2.0.0
    • Component/s: Spark Core
    • Labels:
      None

      Description

      PipeRDD creates a process to run the command and spawns a thread to feed the input data to the process's stdin. If there is any exception in the child thread which gets the input data from the parent RDD, the child thread does not propagate that exception to the main thread. eg. In event of fetch failures, since the exception is not be propagated, the entire stage fails. The correct behaviour would be to recompute the parent(s) and then relaunch the stage.

        Attachments

          Activity

            People

            • Assignee:
              tejasp Tejas Patil
              Reporter:
              tejasp Tejas Patil
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: