Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-21181

Suppress memory leak errors reported by netty

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.1.0
    • Fix Version/s: 2.1.2, 2.2.0, 2.3.0
    • Component/s: Input/Output
    • Labels:
      None

      Description

      We are seeing netty report memory leak erros like the one below after switching to 2.1.

      ERROR ResourceLeakDetector: LEAK: ByteBuf.release() was not called before it's garbage-collected. Enable advanced leak reporting to find out where the leak occurred. To enable advanced leak reporting, specify the JVM option '-Dio.netty.leakDetection.level=advanced' or call ResourceLeakDetector.setLevel() See http://netty.io/wiki/reference-counted-objects.html for more information.
      

      Looking a bit deeper, Spark is not leaking any memory here, but it is confusing for the user to see the error message in the driver logs.

      After enabling, '-Dio.netty.leakDetection.level=advanced', netty reveals the SparkSaslServer to be the source of these leaks.

      Sample trace :https://gist.github.com/dhruve/b299ebc35aa0a185c244a0468927daf1

        Attachments

          Activity

            People

            • Assignee:
              Dhruve Ashar Dhruve Ashar
              Reporter:
              Dhruve Ashar Dhruve Ashar
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: