Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-4567

Connect Producer and Consumer ignore ssl parameters configured for worker

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 0.10.1.1
    • Fix Version/s: 0.11.0.0
    • Component/s: KafkaConnect
    • Labels:
      None

      Description

      When using Connect with a SSL enabled Kafka cluster, the configuration options are either documented a bit misleading, or handled in an incorrect way.

      The documentation states the usual available SSL options (ssl.keystore.location, ssl.truststore.location, ...) and these are picked up and used for the producers and consumers that are used to communicate with the status, offset and configs topics.
      For the producers and consumers that are used for the actual data, these parameters are ignored as can be seen here, which results in plaintext communication on an SSL port, leading to an OOM exception (KAFKA-4493).

      So in order to get Connect to communicate with a secured cluster you need to override all SSL configs with the prefixes consumer. and producer. and duplicate the values already set at a global level.

      The documentation states:

      The most critical site-specific options, such as the Kafka bootstrap servers, are already exposed via the standard worker configuration.

      Since the address for the cluster is exposed here, I would propose that there is no reason not to also pass the SSL parameters through to the consumers and producers, as it is clearly intended that communication happens with the same cluster.
      In fringe cases, these can still be overridden manually to achieve different behavior.

      I am happy to create a pull request to address this or clarify the docs, after we decide which one is the appropriate course of action.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                sliebau Sönke Liebau
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: