Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-6142

Connect worker configurations and connector configurations should accept environment variables

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.0.0
    • Fix Version/s: None
    • Component/s: KafkaConnect
    • Labels:

      Description

      Currently, when a worker or connector configuration is parsed, the values are used as-is without any kind of pre-processing before the value is used. It should be possible to define configuration properties such that string literal values or default values can use configuration variables that reference environment variables and/or system properties, and that these configuration variables are resolved/replaced before the configuration value is used.

      I propose doing this enhancement in Kafka client's ConfigDef by adding a ConfigDef.Transformer interface:

          /**
           * Transform the configuration value.
           */
          public interface Transformer {
              /**
               * Transform the configuration value.
               * @param name The name of the configuration
               * @param value The value of the configuration
               * @return the preprocessed value
               * @throws ConfigException if the value is invalid.
               */
              Object apply(String name, Object value);
          }
      

      and then allowing Transformer implementations to be passed to ConfigDef.define(...) such all existing signatures are maintained for backward compatibility. By default, the definition would use an identity transform that simply returns the value. The transformers would be called in ConfigDef.parseValue(...) before the parseType(...) method is called, and would also be called on the default value if one is provided.

      Then, a ConfigDef.ReplaceSystemVariables implementation would be provided to look in String values for zero or more variables defined with this EBNF grammar:

      '$' '{' varName { ',' varName } [ ':' [ defaultValue] ] '}'
      
      varName ::=  varChar+
      defaultValue ::= varChar*
      varChar ::= any character excluding ',', ':', '}', and '\' except when escaped
      

      where:

      • varName is the name of a Java system property or env. followed by the name of an environment variable, and
      • defaultValue specifies the replacement value used when no environment variable or system property is found, and defaults to an empty string if the `:` is included in the variable

      The value of the first system property or environment variable resolved is then used to replace the variable expression. This implementation would have trace or debug level logging to describe what it is doing.

      Here are several examples:

      Variable Variable replaced with
      ${env.KAFKA_HOME} the value of the KAFKA_HOME environment variable or a null value if that variable doesn't exist
      ${foo.prop1,foo.prop2,env.MY_ENV_VAR:value} the value of the foo.prop1 system property if it exists, or with the value of the foo.prop2 system property if it exists, or with the value of the MY_ENV_VAR environment variable if it exists, or value if none of the system properties exist
      ${foo.prop1,foo.prop2,env.MY_ENV_VAR:} the value of the foo.prop1 system property if it exists, or with the value of the foo.prop2 system property if it exists, or with the value of the MY_ENV_VAR environment variable if it exists, or an empty string if none of the system properties exist
      ${foo.prop1,foo.prop2,env.MY_ENV_VAR} the value of the foo.prop1 system property if it exists, or with the value of the foo.prop2 system property if it exists, or with the value of the MY_ENV_VAR environment variable if it exists, or a null value if none of the system properties exist

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              rhauch Randall Hauch
            • Votes:
              2 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated: