Kafka
  1. Kafka
  2. KAFKA-599

SimpleConsumerShell ONLY connects to the first host in the broker-list string to fetch topic metadata

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Critical Critical
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Activity

      Hide
      John Fung added a comment - - edited
        • This is showing in testcase_0253 in which the first broker (out of the 3-brokers cluster) is down. SimpleConsumerShell doesn't seem to try to connect to other available brokers to fetch topic metadata.

      1. System Test output showing the command line arguments of SimpleConsumerShell

      2012-11-02 06:52:15,728 - INFO - starting debug consumer for replica on [host_0997:9091] partition [0] (kafka_system_test_utils)
      2012-11-02 06:52:15,729 - DEBUG - executing command: [ssh host_0996.prod 'JAVA_HOME=/export/apps/jdk/JDK-1_6_0_21 /kafka_08_replication_system_test/bin/kafka-run-class.sh kafka.tools.SimpleConsumerShell --broker-list host_0997:9091,host_0998:9092,host_0999:9093 --topic test_1 --partition 0 --replica 1 --offset -2 --no-wait-at-logend

      2. SimpleConsumerShell keeps trying on the first host to fetch topic metadata:

      [2012-11-02 06:52:16,290] INFO Getting topic metatdata... (kafka.tools.SimpleConsumerShell$)
      [2012-11-02 06:52:16,352] INFO Fetching metadata for topic Set(test_1) (kafka.client.ClientUtils$)
      [2012-11-02 06:52:16,361] ERROR Producer connection to host_0997:9091 unsuccessful (kafka.producer.SyncProducer)
      java.net.ConnectException: Connection refused
      at sun.nio.ch.Net.connect(Native Method)
      at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
      at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
      at kafka.producer.SyncProducer.connect(SyncProducer.scala:135)
      at kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:150)
      at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:71)
      at kafka.producer.SyncProducer.send(SyncProducer.scala:101)
      at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:25)
      at kafka.tools.SimpleConsumerShell$.main(SimpleConsumerShell.scala:128)
      at kafka.tools.SimpleConsumerShell.main(SimpleConsumerShell.scala)
      [2012-11-02 06:52:16,364] WARN fetching topic metadata for topics [Set(test_1)] from broker [id:0,creatorId:host_0997-1351839136304,host:host_0997,port:9091] failed (kafka.client.ClientUtils$)
      java.net.ConnectException: Connection refused
      at sun.nio.ch.Net.connect(Native Method)
      at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
      at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
      at kafka.producer.SyncProducer.connect(SyncProducer.scala:135)
      at kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:150)
      at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:71)
      at kafka.producer.SyncProducer.send(SyncProducer.scala:101)
      at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:25)
      at kafka.tools.SimpleConsumerShell$.main(SimpleConsumerShell.scala:128)
      at kafka.tools.SimpleConsumerShell.main(SimpleConsumerShell.scala)
      [2012-11-02 06:52:16,364] INFO Fetching metadata for topic Set(test_1) (kafka.client.ClientUtils$)
      [2012-11-02 06:52:16,366] INFO Connected to host_0998:9092 for producing (kafka.producer.SyncProducer)
      [2012-11-02 06:52:16,412] INFO Disconnecting from host_0998:9092 (kafka.producer.SyncProducer)
      [2012-11-02 06:52:18,324] INFO Getting topic metatdata... (kafka.tools.SimpleConsumerShell$)
      [2012-11-02 06:52:18,386] INFO Fetching metadata for topic Set(test_1) (kafka.client.ClientUtils$)
      [2012-11-02 06:52:18,395] ERROR Producer connection to host_0997:9091 unsuccessful (kafka.producer.SyncProducer)
      java.net.ConnectException: Connection refused
      at sun.nio.ch.Net.connect(Native Method)
      at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
      at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
      at kafka.producer.SyncProducer.connect(SyncProducer.scala:135)
      at kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:150)
      at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:71)
      at kafka.producer.SyncProducer.send(SyncProducer.scala:101)
      at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:25)
      at kafka.tools.SimpleConsumerShell$.main(SimpleConsumerShell.scala:128)
      at kafka.tools.SimpleConsumerShell.main(SimpleConsumerShell.scala)
      [2012-11-02 06:52:18,398] WARN fetching topic metadata for topics [Set(test_1)] from broker [id:0,creatorId:host_0997-1351839138338,host:host_0997,port:9091] failed (kafka.client.ClientUtils$)
      java.net.ConnectException: Connection refused
      at sun.nio.ch.Net.connect(Native Method)
      at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
      at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
      at kafka.producer.SyncProducer.connect(SyncProducer.scala:135)
      at kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:150)
      at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:71)
      at kafka.producer.SyncProducer.send(SyncProducer.scala:101)
      at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:25)
      at kafka.tools.SimpleConsumerShell$.main(SimpleConsumerShell.scala:128)
      at kafka.tools.SimpleConsumerShell.main(SimpleConsumerShell.scala)
      [2012-11-02 06:52:18,399] INFO Fetching metadata for topic Set(test_1) (kafka.client.ClientUtils$)
      [2012-11-02 06:52:18,401] INFO Connected to host_0998:9092 for producing (kafka.producer.SyncProducer)
      [2012-11-02 06:52:18,446] INFO Disconnecting from host_0998:9092 (kafka.producer.SyncProducer)
      [2012-11-02 06:52:20,355] INFO Getting topic metatdata... (kafka.tools.SimpleConsumerShell$)
      [2012-11-02 06:52:20,417] INFO Fetching metadata for topic Set(test_1) (kafka.client.ClientUtils$)
      [2012-11-02 06:52:20,426] ERROR Producer connection to host_0997:9091 unsuccessful (kafka.producer.SyncProducer)
      java.net.ConnectException: Connection refused
      at sun.nio.ch.Net.connect(Native Method)
      at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
      at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)

      Show
      John Fung added a comment - - edited This is showing in testcase_0253 in which the first broker (out of the 3-brokers cluster) is down. SimpleConsumerShell doesn't seem to try to connect to other available brokers to fetch topic metadata. 1. System Test output showing the command line arguments of SimpleConsumerShell 2012-11-02 06:52:15,728 - INFO - starting debug consumer for replica on [host_0997:9091] partition [0] (kafka_system_test_utils) 2012-11-02 06:52:15,729 - DEBUG - executing command: [ssh host_0996.prod 'JAVA_HOME=/export/apps/jdk/JDK-1_6_0_21 /kafka_08_replication_system_test/bin/kafka-run-class.sh kafka.tools.SimpleConsumerShell --broker-list host_0997:9091,host_0998:9092,host_0999:9093 --topic test_1 --partition 0 --replica 1 --offset -2 --no-wait-at-logend 2. SimpleConsumerShell keeps trying on the first host to fetch topic metadata: [2012-11-02 06:52:16,290] INFO Getting topic metatdata... (kafka.tools.SimpleConsumerShell$) [2012-11-02 06:52:16,352] INFO Fetching metadata for topic Set(test_1) (kafka.client.ClientUtils$) [2012-11-02 06:52:16,361] ERROR Producer connection to host_0997:9091 unsuccessful (kafka.producer.SyncProducer) java.net.ConnectException: Connection refused at sun.nio.ch.Net.connect(Native Method) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507) at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57) at kafka.producer.SyncProducer.connect(SyncProducer.scala:135) at kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:150) at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:71) at kafka.producer.SyncProducer.send(SyncProducer.scala:101) at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:25) at kafka.tools.SimpleConsumerShell$.main(SimpleConsumerShell.scala:128) at kafka.tools.SimpleConsumerShell.main(SimpleConsumerShell.scala) [2012-11-02 06:52:16,364] WARN fetching topic metadata for topics [Set(test_1)] from broker [id:0,creatorId:host_0997-1351839136304,host:host_0997,port:9091] failed (kafka.client.ClientUtils$) java.net.ConnectException: Connection refused at sun.nio.ch.Net.connect(Native Method) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507) at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57) at kafka.producer.SyncProducer.connect(SyncProducer.scala:135) at kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:150) at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:71) at kafka.producer.SyncProducer.send(SyncProducer.scala:101) at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:25) at kafka.tools.SimpleConsumerShell$.main(SimpleConsumerShell.scala:128) at kafka.tools.SimpleConsumerShell.main(SimpleConsumerShell.scala) [2012-11-02 06:52:16,364] INFO Fetching metadata for topic Set(test_1) (kafka.client.ClientUtils$) [2012-11-02 06:52:16,366] INFO Connected to host_0998:9092 for producing (kafka.producer.SyncProducer) [2012-11-02 06:52:16,412] INFO Disconnecting from host_0998:9092 (kafka.producer.SyncProducer) [2012-11-02 06:52:18,324] INFO Getting topic metatdata... (kafka.tools.SimpleConsumerShell$) [2012-11-02 06:52:18,386] INFO Fetching metadata for topic Set(test_1) (kafka.client.ClientUtils$) [2012-11-02 06:52:18,395] ERROR Producer connection to host_0997:9091 unsuccessful (kafka.producer.SyncProducer) java.net.ConnectException: Connection refused at sun.nio.ch.Net.connect(Native Method) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507) at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57) at kafka.producer.SyncProducer.connect(SyncProducer.scala:135) at kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:150) at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:71) at kafka.producer.SyncProducer.send(SyncProducer.scala:101) at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:25) at kafka.tools.SimpleConsumerShell$.main(SimpleConsumerShell.scala:128) at kafka.tools.SimpleConsumerShell.main(SimpleConsumerShell.scala) [2012-11-02 06:52:18,398] WARN fetching topic metadata for topics [Set(test_1)] from broker [id:0,creatorId:host_0997-1351839138338,host:host_0997,port:9091] failed (kafka.client.ClientUtils$) java.net.ConnectException: Connection refused at sun.nio.ch.Net.connect(Native Method) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507) at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57) at kafka.producer.SyncProducer.connect(SyncProducer.scala:135) at kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:150) at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:71) at kafka.producer.SyncProducer.send(SyncProducer.scala:101) at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:25) at kafka.tools.SimpleConsumerShell$.main(SimpleConsumerShell.scala:128) at kafka.tools.SimpleConsumerShell.main(SimpleConsumerShell.scala) [2012-11-02 06:52:18,399] INFO Fetching metadata for topic Set(test_1) (kafka.client.ClientUtils$) [2012-11-02 06:52:18,401] INFO Connected to host_0998:9092 for producing (kafka.producer.SyncProducer) [2012-11-02 06:52:18,446] INFO Disconnecting from host_0998:9092 (kafka.producer.SyncProducer) [2012-11-02 06:52:20,355] INFO Getting topic metatdata... (kafka.tools.SimpleConsumerShell$) [2012-11-02 06:52:20,417] INFO Fetching metadata for topic Set(test_1) (kafka.client.ClientUtils$) [2012-11-02 06:52:20,426] ERROR Producer connection to host_0997:9091 unsuccessful (kafka.producer.SyncProducer) java.net.ConnectException: Connection refused at sun.nio.ch.Net.connect(Native Method) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507) at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
      John Fung created issue -

        People

        • Assignee:
          Unassigned
          Reporter:
          John Fung
        • Votes:
          0 Vote for this issue
          Watchers:
          1 Start watching this issue

          Dates

          • Created:
            Updated:

            Development