Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-32664

Getting local shuffle block clutters the executor logs

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Trivial
    • Resolution: Fixed
    • Affects Version/s: 3.1.0
    • Fix Version/s: 3.1.0
    • Component/s: Shuffle
    • Labels:
      None

      Description

      The below log statement in BlockManager.getLocalBlockData should be at debug level

      logInfo(s"Getting local shuffle block ${blockId}")
      

      Currently, the executor logs get cluttered with this

      20/08/20 02:07:52 INFO storage.BlockManager: Getting local shuffle block shuffle_0_6103_4964
      20/08/20 02:07:52 INFO storage.BlockManager: Getting local shuffle block shuffle_0_6132_4964
      20/08/20 02:07:52 INFO storage.BlockManager: Getting local shuffle block shuffle_0_6137_4964
      20/08/20 02:07:52 INFO storage.BlockManager: Getting local shuffle block shuffle_0_6312_4964
      20/08/20 02:07:52 INFO storage.BlockManager: Getting local shuffle block shuffle_0_6323_4964
      20/08/20 02:07:52 INFO storage.BlockManager: Getting local shuffle block shuffle_0_6402_4964
      20/08/20 02:07:52 INFO storage.BlockManager: Getting local shuffle block shuffle_0_6413_4964
      20/08/20 02:07:52 INFO storage.BlockManager: Getting local shuffle block shuffle_0_6694_4964
      20/08/20 02:07:52 INFO storage.BlockManager: Getting local shuffle block shuffle_0_6709_4964
      20/08/20 02:07:52 INFO storage.BlockManager: Getting local shuffle block shuffle_0_6753_4964
      20/08/20 02:07:52 INFO storage.BlockManager: Getting local shuffle block shuffle_0_6822_4964
      20/08/20 02:07:52 INFO storage.BlockManager: Getting local shuffle block shuffle_0_6894_4964
      20/08/20 02:07:52 INFO storage.BlockManager: Getting local shuffle block shuffle_0_6913_4964
      20/08/20 02:07:52 INFO storage.BlockManager: Getting local shuffle block shuffle_0_7052_4964
      20/08/20 02:07:52 INFO storage.BlockManager: Getting local shuffle block shuffle_0_7073_4964
      20/08/20 02:07:52 INFO storage.BlockManager: Getting local shuffle block shuffle_0_7167_4964
      20/08/20 02:07:52 INFO storage.BlockManager: Getting local shuffle block shuffle_0_7194_4964
      

      This was added with SPARK-20629.
      cc. Holden Karau

        Attachments

          Activity

            People

            • Assignee:
              dmoore62 Daniel Moore
              Reporter:
              csingh Chandni Singh

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment