Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-17231

Avoid building debug or trace log messages unless the respective log level is enabled

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete CommentsDelete
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 2.0.0
    • 2.0.1, 2.1.0
    • Spark Core
    • None
    • Spark cluster with 8 r3.8xl EC2 worker instances

    Description

      While debugging the performance of a large GraphX connected components computation, I found several places in the network-common and network-shuffle code bases where trace or debug log messages are constructed even if the respective log level is disabled. Refactoring the respective code to avoid these constructions except where necessary led to a modest but measurable reduction in task time, GC time and the ratio thereof.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            michael Michael MacFadden Assign to me
            michael Michael MacFadden
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment