Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-2221

Make KafkaIO coder specification less awkward

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.0
    • Component/s: io-java-kafka
    • Labels:
      None

      Description

      readWithCoders and writeWithCoders functions are awkward because they don't emphasize enough that coders are a poor choice for interpreting wire format.

      The only reason to specify coders in KafkaIO is when coder inference from Deserializer fails. To emphasize that, let's change the API to be withKeyDeserializer(Deserializer) as the default choice and withKeyDeserializerAndCoder(Deserializer,Coder) if inference fails; likewise for value.

      Remove functions using coders to interpret wire format from the API. A common case of that is Avro and Proto - for that, introduce special helper functions, I guess like withAvro/ProtoKey/Value(...), which under the hood may be allowed to reuse Avro/ProtoCoder as a utility, but do not expose this fact.

        Attachments

          Activity

            People

            • Assignee:
              rangadi Raghu Angadi
              Reporter:
              jkff Eugene Kirpichov
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: