[SPARK-32664] Getting local shuffle block clutters the executor logs

• Type: Bug
• Status: Resolved
• Priority: Trivial
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Fix Version/s: 3.1.0
• Component/s: Shuffle
• Assignee: Daniel Moore
• Reporter: Chandni Singh
• Created: 20/Aug/20 06:05
• Updated: 12/Dec/22 18:11
• Resolved: 25/Aug/20 13:47

Description

The below log statement in BlockManager.getLocalBlockData should be at debug level

logInfo(s"Getting local shuffle block ${blockId}")

Currently, the executor logs get cluttered with this

20/08/20 02:07:52 INFO storage.BlockManager: Getting local shuffle block shuffle_0_6103_4964
20/08/20 02:07:52 INFO storage.BlockManager: Getting local shuffle block shuffle_0_6132_4964
20/08/20 02:07:52 INFO storage.BlockManager: Getting local shuffle block shuffle_0_6137_4964
20/08/20 02:07:52 INFO storage.BlockManager: Getting local shuffle block shuffle_0_6312_4964
20/08/20 02:07:52 INFO storage.BlockManager: Getting local shuffle block shuffle_0_6323_4964
20/08/20 02:07:52 INFO storage.BlockManager: Getting local shuffle block shuffle_0_6402_4964
20/08/20 02:07:52 INFO storage.BlockManager: Getting local shuffle block shuffle_0_6413_4964
20/08/20 02:07:52 INFO storage.BlockManager: Getting local shuffle block shuffle_0_6694_4964
20/08/20 02:07:52 INFO storage.BlockManager: Getting local shuffle block shuffle_0_6709_4964
20/08/20 02:07:52 INFO storage.BlockManager: Getting local shuffle block shuffle_0_6753_4964
20/08/20 02:07:52 INFO storage.BlockManager: Getting local shuffle block shuffle_0_6822_4964
20/08/20 02:07:52 INFO storage.BlockManager: Getting local shuffle block shuffle_0_6894_4964
20/08/20 02:07:52 INFO storage.BlockManager: Getting local shuffle block shuffle_0_6913_4964
20/08/20 02:07:52 INFO storage.BlockManager: Getting local shuffle block shuffle_0_7052_4964
20/08/20 02:07:52 INFO storage.BlockManager: Getting local shuffle block shuffle_0_7073_4964
20/08/20 02:07:52 INFO storage.BlockManager: Getting local shuffle block shuffle_0_7167_4964
20/08/20 02:07:52 INFO storage.BlockManager: Getting local shuffle block shuffle_0_7194_4964

This was added with SPARK-20629.
cc. holden

Issue Links

• links to: [Github] Pull Request #29527 (dmoore62)