Uploaded image for project: 'Kudu'
  1. Kudu
  2. KUDU-1453

Spark executors leak kudu clients and netty threads

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • 0.8.0
    • 0.9.0
    • spark
    • None

    Description

      On a test cluster, every time I run a spark SQL query against a table, each of my spark worker tasks end up with another ~500 netty worker threads created. It seems like each Spark partition/task is creating its own KuduClient which then creates a bunch of worker threads and never cleans them up.

      I'm calling this a blocker since after ~20 queries or so, the machines woudl run out of threads and crash.

      Attachments

        Activity

          People

            bgard6977 Brent Gardner
            tlipcon Todd Lipcon
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: