Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-21321

Spark very verbose on shutdown confusing users

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.1.1
    • Fix Version/s: 2.2.1, 2.3.0
    • Component/s: Spark Core
    • Labels:
      None

      Description

      On shutdown, spark can be very verbose and can spit out errors that causes the user to be confused.

      If possible, we should not print those out and just ignore them by changing the changing the log level from WARNING to DEBUG.

      Also shutdown of spark can take a long time because of the backlog of events in the event queue ( warnings like: Message $message dropped. ${e.getMessage}).

      This happens more with dynamic allocation on.

      I am suggesting to
      1. change the log level when the shutdown is happening and the RPC connections are closed(RpcEnvStoppedException).

      2. Clear the event queue when RPC module is stopped and spark is shutting down.

        Attachments

          Activity

            People

            • Assignee:
              yoonlee95 Jong Yoon Lee
              Reporter:
              yoonlee95 Jong Yoon Lee
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 24h
                24h
                Remaining:
                Remaining Estimate - 24h
                24h
                Logged:
                Time Spent - Not Specified
                Not Specified