Kafka
  1. Kafka
  2. KAFKA-1092

Add server config parameter to separate bind address and ZK hostname

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.1
    • Fix Version/s: 0.8.1
    • Component/s: config
    • Labels:
      None

      Description

      Currently, in server.properties, you can configure host.name which gets used for two purposes: 1) to bind the socket 2) to publish the broker details to ZK for clients to use.

      There are times when these two settings need to be different. Here's an example. I want to setup Kafka brokers on OpenStack virtual machines in a private cloud but I need producers to connect from elsewhere on the internal corporate network. With OpenStack, the virtual machines are only exposed to DHCP addresses (typically RFC 1918 private addresses). You can assign "floating ips" to a virtual machine but it's forwarded using Network Address Translation and not exposed directly to the VM. Also, there's typically no DNS to provide hostname lookup. Hosts have names like "fubar.novalocal" that are not externally routable.

      Here's what I want. I want the broker to bind to the VM's private network IP but I want it to publish it's floating IP to ZooKeeper so that producers can publish to it.

      I propose a new optional parameter, "listen", which would allow you to specify the socket address to listen on. If not set, the parameter would default to host.name, which is the current behavior.

      #Publish the externally routable IP in ZK
      host.name = <floating ip>
      #Accept connections from any interface the VM knows about
      listen = *

      1. KAFKA-1092.patch
        29 kB
        Roger Hoover
      2. KAFKA-1092.patch
        18 kB
        Roger Hoover

        Activity

        Hide
        Roger Hoover added a comment - - edited

        Added two new configuration parameters:

        advertise.host.name - Hostname the broker will advertise to producers and consumers. If not set, it uses the value for "host.name" if configured. Otherwise, it will use the value returned from java.net.InetAddress.getCanonicalHostName().

        advertise.port - The port to publish to ZooKeeper for clients to use. If this is not set, it will publish the same port that the broker binds to.

        Show
        Roger Hoover added a comment - - edited Added two new configuration parameters: advertise.host.name - Hostname the broker will advertise to producers and consumers. If not set, it uses the value for "host.name" if configured. Otherwise, it will use the value returned from java.net.InetAddress.getCanonicalHostName(). advertise.port - The port to publish to ZooKeeper for clients to use. If this is not set, it will publish the same port that the broker binds to.
        Hide
        Roger Hoover added a comment -

        Note: this change is backward compatible. Not specifying the two new optional parameters will yield the same behavior as before

        Show
        Roger Hoover added a comment - Note: this change is backward compatible. Not specifying the two new optional parameters will yield the same behavior as before
        Hide
        Jun Rao added a comment -

        Thanks for the patch.

        1. Could you change the property name to advertised.host.name and advertised.port?

        2. All unit tests in kafka.producer.ProducerTest seem to fail with this patch.

        Show
        Jun Rao added a comment - Thanks for the patch. 1. Could you change the property name to advertised.host.name and advertised.port? 2. All unit tests in kafka.producer.ProducerTest seem to fail with this patch.
        Hide
        Roger Hoover added a comment -

        Updated parameter names to "advertised.host.name" and "advertised.port" and fixed broken ProducerTest

        Show
        Roger Hoover added a comment - Updated parameter names to "advertised.host.name" and "advertised.port" and fixed broken ProducerTest
        Hide
        Roger Hoover added a comment - - edited

        The reason the kafka.producer.ProducerTest was broken was because that test was creating an anonymous subclass of KafkaConfig and overriding a couple of members. I don't know Scala that well but discovered that statically initialized variables that depend on other statically initialized variables behave unexpectedly. To fix this, I just set properties in the test and do not subclass KafkaConfig, as users would do.

        class A() {
             val x = 1
             val y = x
        }
        
        class B() extends A {
             override val x = 2
        }
        
        val b = new B()
        b.y //this is 0, not 1 or 2
        
        Show
        Roger Hoover added a comment - - edited The reason the kafka.producer.ProducerTest was broken was because that test was creating an anonymous subclass of KafkaConfig and overriding a couple of members. I don't know Scala that well but discovered that statically initialized variables that depend on other statically initialized variables behave unexpectedly. To fix this, I just set properties in the test and do not subclass KafkaConfig, as users would do. class A() { val x = 1 val y = x } class B() extends A { override val x = 2 } val b = new B() b.y // this is 0, not 1 or 2
        Hide
        Jun Rao added a comment -

        Thanks for the patch. +1. Committed to trunk after adding the ASF header to the new test file.

        Show
        Jun Rao added a comment - Thanks for the patch. +1. Committed to trunk after adding the ASF header to the new test file.

          People

          • Assignee:
            Roger Hoover
            Reporter:
            Roger Hoover
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development