Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-17231

Avoid building debug or trace log messages unless the respective log level is enabled

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 2.0.0
    • 2.0.1, 2.1.0
    • Spark Core
    • None
    • Spark cluster with 8 r3.8xl EC2 worker instances

    Description

      While debugging the performance of a large GraphX connected components computation, I found several places in the network-common and network-shuffle code bases where trace or debug log messages are constructed even if the respective log level is disabled. Refactoring the respective code to avoid these constructions except where necessary led to a modest but measurable reduction in task time, GC time and the ratio thereof.

      Attachments

        1. logging_perf_improvements.jpg
          63 kB
          Michael MacFadden
        2. logging_perf_improvements 2.jpg
          66 kB
          Michael MacFadden
        3. master.jpg
          63 kB
          Michael MacFadden
        4. master 2.jpg
          66 kB
          Michael MacFadden

        Activity

          People

            michael Michael MacFadden
            michael Michael MacFadden
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: