Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-21490

SparkLauncher may fail to redirect streams

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.2.0
    • Fix Version/s: 2.3.0
    • Component/s: Spark Core
    • Labels:
      None

      Description

      While investigating a different issue, I noticed that the redirection handling in SparkLauncher is faulty.

      In the default case or when users use only log redirection, things should work fine.

      But if users try to redirect just the stdout of a child process without redirecting stderr, the launcher won't detect that and the child process may hang because stderr is not being read and its buffer fills up.

      The code should detect these cases and redirect any streams that haven't been explicitly redirected by the user.

        Attachments

          Activity

            People

            • Assignee:
              vanzin Marcelo Vanzin
              Reporter:
              vanzin Marcelo Vanzin
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: