Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-2034

KafkaInputDStream doesn't close resources and may prevent JVM shutdown

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: 1.0.1, 1.1.0
    • Component/s: DStreams
    • Labels:
      None

      Description

      Tobias noted today on the mailing list:

      I am trying to use Spark Streaming with Kafka, which works like a
      charm – except for shutdown. When I run my program with "sbt
      run-main", sbt will never exit, because there are two non-daemon
      threads left that don't die.

      I created a minimal example at
      <https://gist.github.com/tgpfeiffer/b1e765064e983449c6b6#file-kafkadoesntshutdown-scala>.
      It starts a StreamingContext and does nothing more than connecting to
      a Kafka server and printing what it receives. Using the `future { ...
      }` construct, I shut down the StreamingContext after some seconds and
      then print the difference between the threads at start time and at end
      time. The output can be found at
      <https://gist.github.com/tgpfeiffer/b1e765064e983449c6b6#file-output1>.
      There are a number of threads remaining that will prevent sbt from
      exiting.

      When I replace `KafkaUtils.createStream(...)` with a call that does
      exactly the same, except that it calls `consumerConnector.shutdown()`
      in `KafkaReceiver.onStop()` (which it should, IMO), the output is as
      shown at <https://gist.github.com/tgpfeiffer/b1e765064e983449c6b6#file-output2>.

      Does anyone have any idea what is going on here and why the program
      doesn't shut down properly? The behavior is the same with both kafka
      0.8.0 and 0.8.1.1, by the way.

      Something similar was noted last year:

      http://mail-archives.apache.org/mod_mbox/spark-dev/201309.mbox/%3C1380220041.2428.YahooMailNeo@web160804.mail.bf1.yahoo.com%3E

      KafkaInputDStream doesn't close ConsumerConnector in onStop(), and does not close the Executor it creates. The latter leaves non-daemon threads and can prevent the JVM from shutting down even if streaming is closed properly.

        Attachments

          Activity

            People

            • Assignee:
              srowen Sean R. Owen
              Reporter:
              srowen Sean R. Owen
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: