[SPARK-23055] KafkaContinuousSourceSuite Kafka column types test failing

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 2.3.0
• Component/s: Structured Streaming
• Target Version/s: 2.3.0
• Assignee: Sameer Agarwal
• Reporter: Marco Gaido
• Created: 12/Jan/18 08:26
• Updated: 12/Jan/18 23:41
• Resolved: 12/Jan/18 23:21

Description

KafkaContinuousSourceSuite Kafka column types test fails (https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85998/testReport/junit/org.apache.spark.sql.kafka010/KafkaContinuousSourceSuite/Kafka_column_types/).

https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7/4009/

[info] KafkaContinuousSourceSuite:
[info] - cannot stop Kafka stream (1 second, 279 milliseconds)
[info] - assign from latest offsets (failOnDataLoss: true) (9 seconds, 202 milliseconds)
[info] - assign from earliest offsets (failOnDataLoss: true) (8 seconds, 108 milliseconds)
[info] - assign from specific offsets (failOnDataLoss: true) (4 seconds, 102 milliseconds)
[info] - subscribing topic by name from latest offsets (failOnDataLoss: true) (12 seconds, 125 milliseconds)
[info] - subscribing topic by name from earliest offsets (failOnDataLoss: true) (12 seconds, 69 milliseconds)
[info] - subscribing topic by name from specific offsets (failOnDataLoss: true) (5 seconds, 935 milliseconds)
[info] - subscribing topic by pattern from latest offsets (failOnDataLoss: true) (13 seconds, 70 milliseconds)
[info] - subscribing topic by pattern from earliest offsets (failOnDataLoss: true) (13 seconds, 122 milliseconds)
[info] - subscribing topic by pattern from specific offsets (failOnDataLoss: true) (7 seconds, 877 milliseconds)
[info] - assign from latest offsets (failOnDataLoss: false) (12 seconds, 201 milliseconds)
[info] - assign from earliest offsets (failOnDataLoss: false) (12 seconds, 82 milliseconds)
[info] - assign from specific offsets (failOnDataLoss: false) (8 seconds, 530 milliseconds)
[info] - subscribing topic by name from latest offsets (failOnDataLoss: false) (18 seconds, 339 milliseconds)
[info] - subscribing topic by name from earliest offsets (failOnDataLoss: false) (17 seconds, 397 milliseconds)
[info] - subscribing topic by name from specific offsets (failOnDataLoss: false) (8 seconds, 926 milliseconds)
[info] - subscribing topic by pattern from latest offsets (failOnDataLoss: false) (20 seconds, 198 milliseconds)
Build timed out (after 255 minutes). Marking the build as aborted.
Build was aborted
Archiving artifacts
[info] - subscribing topic by pattern from earliest offsets (failOnDataLoss: false) *** FAILED *** (2 hours, 24 minutes, 19 seconds)
[info]   Error while stopping stream: 

https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7/4010/console

[info] KafkaContinuousSourceSuite:
[info] - cannot stop Kafka stream (1 second, 238 milliseconds)
[info] - assign from latest offsets (failOnDataLoss: true) (9 seconds, 516 milliseconds)
[info] - assign from earliest offsets (failOnDataLoss: true) (7 seconds, 961 milliseconds)
[info] - assign from specific offsets (failOnDataLoss: true) (4 seconds, 193 milliseconds)
[info] - subscribing topic by name from latest offsets (failOnDataLoss: true) (11 seconds, 443 milliseconds)
[info] - subscribing topic by name from earliest offsets (failOnDataLoss: true) (12 seconds, 674 milliseconds)
[info] - subscribing topic by name from specific offsets (failOnDataLoss: true) (6 seconds, 13 milliseconds)
[info] - subscribing topic by pattern from latest offsets (failOnDataLoss: true) (13 seconds, 185 milliseconds)
Build timed out (after 255 minutes). Marking the build as aborted.

https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7/4011/consoleFull

[info] KafkaContinuousSourceSuite:
[info] - cannot stop Kafka stream (1 second, 551 milliseconds)
[info] - assign from latest offsets (failOnDataLoss: true) (8 seconds, 560 milliseconds)
[info] - assign from earliest offsets (failOnDataLoss: true) (8 seconds, 40 milliseconds)
[info] - assign from specific offsets (failOnDataLoss: true) (4 seconds, 373 milliseconds)
[info] - subscribing topic by name from latest offsets (failOnDataLoss: true) (12 seconds, 872 milliseconds)
[info] - subscribing topic by name from earliest offsets (failOnDataLoss: true) (13 seconds, 338 milliseconds)
[info] - subscribing topic by name from specific offsets (failOnDataLoss: true) (6 seconds, 999 milliseconds)
[info] - subscribing topic by pattern from latest offsets (failOnDataLoss: true) (15 seconds, 393 milliseconds)
[info] - subscribing topic by pattern from earliest offsets (failOnDataLoss: true) (14 seconds, 482 milliseconds)
Build timed out (after 255 minutes). Marking the build as aborted.
Build was aborted
Archiving artifacts
[info] - subscribing topic by pattern from specific offsets (failOnDataLoss: true) *** FAILED *** (2 hours, 24 minutes, 1 second)
[info]   Error while stopping stream: 
.. tim

Issue Links

• relates to: SPARK-22908 add basic continuous kafka source