[DRILL-6625] Intermittent failures in Kafka unit tests

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.13.0
• Fix Version/s: 1.15.0
• Component/s: Storage - Kafka
• Labels: ready-to-commit
• Assignee: Abhishek Ravi
• Reporter: Boaz Ben-Zvi
• Reviewer: Timothy Farkas
• Created: 21/Jul/18 01:53
• Updated: 15/Sep/18 14:57
• Resolved: 15/Sep/18 14:57

Description

The following failures have been seen (consistently on my Mac, or occasionally on Jenkins) when running the unit tests, in the Kafka test suit. After the failure, maven hangs for a long time.

Cost was 0.0 (instead of 26.0) :

Running org.apache.drill.exec.store.kafka.KafkaFilterPushdownTest
16:46:57.748 [main] ERROR org.apache.drill.TestReporter - Test Failed (d: -65.3 KiB(73.6 KiB), h: -573.5 MiB(379.5 MiB), nh: 1.2 MiB(117.1 MiB)): testPushdownWithOr(org.apache.drill.exec.store.kafka.KafkaFilterPushdownTest)
java.lang.AssertionError: Unable to find expected string     "kafkaScanSpec" : {
      "topicName" : "drill-pushdown-topic"
    },
    "cost" : 26.0 in plan: {
  "head" : {
    "version" : 1,
    "generator" : {
      "type" : "ExplainHandler",
      "info" : ""
    },
    "type" : "APACHE_DRILL_PHYSICAL",
    "options" : [ {
      "kind" : "STRING",
      "accessibleScopes" : "ALL",
      "name" : "store.kafka.record.reader",
      "string_val" : "org.apache.drill.exec.store.kafka.decoders.JsonMessageReader",
      "scope" : "SESSION"
    }, {
      "kind" : "LONG",
      "accessibleScopes" : "ALL",
      "name" : "planner.width.max_per_node",
      "num_val" : 2,
      "scope" : "SESSION"
    }, {
      "kind" : "BOOLEAN",
      "accessibleScopes" : "ALL",
      "name" : "exec.errors.verbose",
      "bool_val" : true,
      "scope" : "SESSION"
    }, {
      "kind" : "LONG",
      "accessibleScopes" : "ALL",
      "name" : "store.kafka.poll.timeout",
      "num_val" : 200,
      "scope" : "SESSION"
    } ],
    "queue" : 0,
    "hasResourcePlan" : false,
    "resultMode" : "EXEC"
  },
  "graph" : [ {
    "pop" : "kafka-scan",
    "@id" : 6,
    "userName" : "",
    "kafkaStoragePluginConfig" : {
      "type" : "kafka",
      "kafkaConsumerProps" : {
        "bootstrap.servers" : "127.0.0.1:63751",
        "group.id" : "drill-test-consumer"
      },
      "enabled" : true
    },
    "columns" : [ "`**`" ],
    "kafkaScanSpec" : {
      "topicName" : "drill-pushdown-topic"
    },
    "cost" : 0.0
  }, {

Or occasionally:

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
11:52:57.571 [main] ERROR o.a.d.e.s.k.KafkaMessageGenerator - org.apache.kafka.common.errors.NetworkException: The server disconnected before a response was received.
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.NetworkException: The server disconnected before a response was received.

Issue Links

• links to: GitHub Pull Request #1463