Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-20624 SPIP: Add better handling for node shutdown
  3. SPARK-36543

Decommission logs too frequent when waiting migration to finish

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsAdd voteVotersStop watchingWatchersConvert to IssueLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: In Progress
    • Major
    • Resolution: Unresolved
    • 3.1.0, 3.2.0, 3.3.0
    • None
    • Spark Core
    • None

    Description

      21/08/18 08:14:31 INFO CoarseGrainedExecutorBackend: Checking to see if we can shutdown.
      21/08/18 08:14:31 INFO CoarseGrainedExecutorBackend: No running tasks, checking migrations
      21/08/18 08:14:31 INFO CoarseGrainedExecutorBackend: All blocks not yet migrated.
      21/08/18 08:14:32 INFO CoarseGrainedExecutorBackend: Checking to see if we can shutdown.
      21/08/18 08:14:32 INFO CoarseGrainedExecutorBackend: No running tasks, checking migrations
      21/08/18 08:14:32 INFO CoarseGrainedExecutorBackend: All blocks not yet migrated.
      21/08/18 08:14:33 INFO CoarseGrainedExecutorBackend: Checking to see if we can shutdown.
      21/08/18 08:14:33 INFO CoarseGrainedExecutorBackend: No running tasks, checking migrations
      21/08/18 08:14:33 INFO CoarseGrainedExecutorBackend: All blocks not yet migrated.
      21/08/18 08:14:34 INFO CoarseGrainedExecutorBackend: Checking to see if we can shutdown.
      21/08/18 08:14:34 INFO CoarseGrainedExecutorBackend: No running tasks, checking migrations
      21/08/18 08:14:34 INFO CoarseGrainedExecutorBackend: All blocks not yet migrated.
      21/08/18 08:14:35 INFO CoarseGrainedExecutorBackend: Checking to see if we can shutdown.
      21/08/18 08:14:35 INFO CoarseGrainedExecutorBackend: No running tasks, checking migrations
      21/08/18 08:14:35 INFO CoarseGrainedExecutorBackend: All blocks not yet migrated.
      21/08/18 08:14:36 INFO CoarseGrainedExecutorBackend: Checking to see if we can shutdown.
      21/08/18 08:14:36 INFO CoarseGrainedExecutorBackend: No running tasks, checking migrations
      21/08/18 08:14:36 INFO CoarseGrainedExecutorBackend: All blocks not yet migrated.
      21/08/18 08:14:37 INFO CoarseGrainedExecutorBackend: Checking to see if we can shutdown.
      21/08/18 08:14:37 INFO CoarseGrainedExecutorBackend: No running tasks, checking migrations
      21/08/18 08:14:37 INFO CoarseGrainedExecutorBackend: All blocks not yet migrated.
      21/08/18 08:14:38 INFO CoarseGrainedExecutorBackend: Checking to see if we can shutdown.
      21/08/18 08:14:38 INFO CoarseGrainedExecutorBackend: No running tasks, checking migrations
      21/08/18 08:14:38 INFO CoarseGrainedExecutorBackend: All blocks not yet migrated.
      21/08/18 08:14:39 INFO CoarseGrainedExecutorBackend: Checking to see if we can shutdown.
      21/08/18 08:14:39 INFO CoarseGrainedExecutorBackend: No running tasks, checking migrations
      21/08/18 08:14:39 INFO CoarseGrainedExecutorBackend: All blocks not yet migrated.
      21/08/18 08:14:40 INFO CoarseGrainedExecutorBackend: Checking to see if we can shutdown.
      21/08/18 08:14:40 INFO CoarseGrainedExecutorBackend: No running tasks, checking migrations
      21/08/18 08:14:40 INFO CoarseGrainedExecutorBackend: All blocks not yet migrated.
      21/08/18 08:14:41 INFO CoarseGrainedExecutorBackend: Checking to see if we can shutdown.
      21/08/18 08:14:41 INFO CoarseGrainedExecutorBackend: No running tasks, checking migrations
      21/08/18 08:14:41 INFO CoarseGrainedExecutorBackend: All blocks not yet migrated.
      21/08/18 08:14:42 INFO CoarseGrainedExecutorBackend: Checking to see if we can shutdown.
      21/08/18 08:14:42 INFO CoarseGrainedExecutorBackend: No running tasks, checking migrations
      21/08/18 08:14:42 INFO CoarseGrainedExecutorBackend: All blocks not yet migrated.
      21/08/18 08:14:43 INFO CoarseGrainedExecutorBackend: Checking to see if we can shutdown.
      21/08/18 08:14:43 INFO CoarseGrainedExecutorBackend: No running tasks, checking migrations
      21/08/18 08:14:43 INFO CoarseGrainedExecutorBackend: All blocks not yet migrated.
      21/08/18 08:14:44 INFO CoarseGrainedExecutorBackend: Checking to see if we can shutdown.
      21/08/18 08:14:44 INFO CoarseGrainedExecutorBackend: No running tasks, checking migrations
      21/08/18 08:14:44 INFO CoarseGrainedExecutorBackend: All blocks not yet migrated.
      ...

      It takes some time to migrate data (shuffle or rdd). Logging per second is too frequent. 

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            apachespark Apache Spark
            Ngone51 wuyi
            Holden Karau Holden Karau

            Dates

              Created:
              Updated:

              Slack

                Issue deployment