Uploaded image for project: 'Apache Hudi'
  1. Apache Hudi
  2. HUDI-3264

Make schema registry configs more flexible with MultiTableDeltaStreamer

    XMLWordPrintableJSON

Details

    Description

      Ref issue: https://github.com/apache/hudi/issues/4585

      Hi guys,

      we ran into a problem setting the target schema of our Hudi table using the MultiTableDeltaStreamer.

      Using a normal DeltaStreamer, we are able to set our source and target schemas using the properties:

      • hoodie.deltastreamer.schemaprovider.registry.url
      • hoodie.deltastreamer.schemaprovider.registry.targetUrl

      We found that we are not able to set these properties on a table basis using the MultiTableDeltaStreamer, since the MTDS builds SchemaRegistry URLs for target and source schema using the properties:

      • hoodie.deltastreamer.schemaprovider.registry.baseUrl
      • hoodie.deltastreamer.schemaprovider.registry.sourceUrlSuffix
      • hoodie.deltastreamer.schemaprovider.registry.targetUrlSuffix

      Later the MultiTableDeltaStreamer uses the source Kafka Topic name also for setting the name of the target schema:

       
      hudi/hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieMultiTableDeltaStreamer.java

      Line 167 in 9fe28e5

      typedProperties.setProperty(Constants.TARGET_SCHEMA_REGISTRY_URL_PROP, schemaRegistryBaseUrl + typedProperties.getString(Constants.KAFKA_TOPIC_PROP) + targetSchemaRegistrySuffix);

       

      We think, that schema names should be more configurable, like the origin DeltaStreamer would handle it. Actually the names of the schemas you want to use for reading or writing the data are very tight coupled to the name of the Kafka topic the data is loaded from.

       

       

       

      Attachments

        Issue Links

          Activity

            People

              Pratyaksh Pratyaksh Sharma
              shivnarayan sivabalan narayanan
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: