Uploaded image for project: 'Apache Gobblin'
  1. Apache Gobblin
  2. GOBBLIN-376

Kafka Schema Registration for RecordWithMetadata and BytesToRecordWithMetadataConverter

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 0.14.0
    • misc
    • None

    Description

      RecordWithMetadataSchemaRegistrationConverter:

      An Avro Schema might need to be registered to Kafka. Using RecordWithMetadata registered schema id can be added to record metadata. Hence adding a converter with takes RecordWithMetadata, registers Avro schema with Kafka schema registry, adds the schema id to metadata and returns RecordWithMetadata will be useful.

       

      BytestoRecordWithMetadata converter:

      To wrap bytes along with metadata

       

      Use case:

      Needed a converter which takes serialized Avro record (byte array) and schema as the string, and returns the record along with the registered schemaId.

      RecordWithMetadata object can be used to carry the schemaId along with the record(in this case byte array). Hence the above two converters in order of BytestoRecordWithMetadata and RecordWithMetadataSchemaRegistrationConverter will do the work.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            adsharma Aditya Sharma
            adsharma Aditya Sharma
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment