[SPARK-36543] Decommission logs too frequent when waiting migration to finish

• Type: Sub-task
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.0, 3.2.0, 3.3.0
• Component/s: Spark Core
• Assignee: Apache Spark
• Reporter: wuyi
• Shepherd: Holden Karau
• Created: 18/Aug/21 12:55
• Updated: 04/Jan/22 09:00

Description

21/08/18 08:14:31 INFO CoarseGrainedExecutorBackend: Checking to see if we can shutdown.
21/08/18 08:14:31 INFO CoarseGrainedExecutorBackend: No running tasks, checking migrations
21/08/18 08:14:31 INFO CoarseGrainedExecutorBackend: All blocks not yet migrated.
21/08/18 08:14:32 INFO CoarseGrainedExecutorBackend: Checking to see if we can shutdown.
21/08/18 08:14:32 INFO CoarseGrainedExecutorBackend: No running tasks, checking migrations
21/08/18 08:14:32 INFO CoarseGrainedExecutorBackend: All blocks not yet migrated.
21/08/18 08:14:33 INFO CoarseGrainedExecutorBackend: Checking to see if we can shutdown.
21/08/18 08:14:33 INFO CoarseGrainedExecutorBackend: No running tasks, checking migrations
21/08/18 08:14:33 INFO CoarseGrainedExecutorBackend: All blocks not yet migrated.
21/08/18 08:14:34 INFO CoarseGrainedExecutorBackend: Checking to see if we can shutdown.
21/08/18 08:14:34 INFO CoarseGrainedExecutorBackend: No running tasks, checking migrations
21/08/18 08:14:34 INFO CoarseGrainedExecutorBackend: All blocks not yet migrated.
21/08/18 08:14:35 INFO CoarseGrainedExecutorBackend: Checking to see if we can shutdown.
21/08/18 08:14:35 INFO CoarseGrainedExecutorBackend: No running tasks, checking migrations
21/08/18 08:14:35 INFO CoarseGrainedExecutorBackend: All blocks not yet migrated.
21/08/18 08:14:36 INFO CoarseGrainedExecutorBackend: Checking to see if we can shutdown.
21/08/18 08:14:36 INFO CoarseGrainedExecutorBackend: No running tasks, checking migrations
21/08/18 08:14:36 INFO CoarseGrainedExecutorBackend: All blocks not yet migrated.
21/08/18 08:14:37 INFO CoarseGrainedExecutorBackend: Checking to see if we can shutdown.
21/08/18 08:14:37 INFO CoarseGrainedExecutorBackend: No running tasks, checking migrations
21/08/18 08:14:37 INFO CoarseGrainedExecutorBackend: All blocks not yet migrated.
21/08/18 08:14:38 INFO CoarseGrainedExecutorBackend: Checking to see if we can shutdown.
21/08/18 08:14:38 INFO CoarseGrainedExecutorBackend: No running tasks, checking migrations
21/08/18 08:14:38 INFO CoarseGrainedExecutorBackend: All blocks not yet migrated.
21/08/18 08:14:39 INFO CoarseGrainedExecutorBackend: Checking to see if we can shutdown.
21/08/18 08:14:39 INFO CoarseGrainedExecutorBackend: No running tasks, checking migrations
21/08/18 08:14:39 INFO CoarseGrainedExecutorBackend: All blocks not yet migrated.
21/08/18 08:14:40 INFO CoarseGrainedExecutorBackend: Checking to see if we can shutdown.
21/08/18 08:14:40 INFO CoarseGrainedExecutorBackend: No running tasks, checking migrations
21/08/18 08:14:40 INFO CoarseGrainedExecutorBackend: All blocks not yet migrated.
21/08/18 08:14:41 INFO CoarseGrainedExecutorBackend: Checking to see if we can shutdown.
21/08/18 08:14:41 INFO CoarseGrainedExecutorBackend: No running tasks, checking migrations
21/08/18 08:14:41 INFO CoarseGrainedExecutorBackend: All blocks not yet migrated.
21/08/18 08:14:42 INFO CoarseGrainedExecutorBackend: Checking to see if we can shutdown.
21/08/18 08:14:42 INFO CoarseGrainedExecutorBackend: No running tasks, checking migrations
21/08/18 08:14:42 INFO CoarseGrainedExecutorBackend: All blocks not yet migrated.
21/08/18 08:14:43 INFO CoarseGrainedExecutorBackend: Checking to see if we can shutdown.
21/08/18 08:14:43 INFO CoarseGrainedExecutorBackend: No running tasks, checking migrations
21/08/18 08:14:43 INFO CoarseGrainedExecutorBackend: All blocks not yet migrated.
21/08/18 08:14:44 INFO CoarseGrainedExecutorBackend: Checking to see if we can shutdown.
21/08/18 08:14:44 INFO CoarseGrainedExecutorBackend: No running tasks, checking migrations
21/08/18 08:14:44 INFO CoarseGrainedExecutorBackend: All blocks not yet migrated.
...

It takes some time to migrate data (shuffle or rdd). Logging per second is too frequent. 

Issue Links

• links to: [Github] Pull Request #35094 (sungpeo)