Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-3921

Utilize schema embedded in incoming FlowFile/Record from record Writers

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.2.0
    • 1.4.0
    • Extensions
    • None

    Description

      Currently, any record writer has to retrieve schema from either SchemaRegistry or Schema text. It doesn't work well with schema embedded Avro record as incoming FlowFile, or used with any schema inferable readers such as CSVReader.

      For example, ExecuteSQL outputs Avro record with schema embedded, and if user wants to convert it to Json, they have to register the schema to SchemaRegistry.

      I expect we don't have to touch registry in that use-case.

      AvroReader has 'Use Embedded Avro Schema' option for 'Schema Access Strategy', so processor such as ConvertRecord or QueryRecord can read record with embedded schema, however since any writer doesn't have such option for 'Schema Access Strategy', user have to register the schema in a SchemaRegistry for those record operations to work properly.

      I had to view a FlowFile which has embedded schema from NiFi content viewer and copy the schema text then add it to registry. Plus extra UpdateAttribute is needed to add 'schema.name' attribute. These extra work can be avoided.

      If writers can utilize embedded schema, we can provide more seamless UX with schema embedded data.

      Probably we can use schema attached to record that Bryan Bende mentioned at nifi-dev ML:

      When a reader produces a record it attaches the schema it used to the
      record, but we currently don't have a way for a writer to use that
      schema when writing a record, although I think we do want to support
      that... something like a "Use Schema in Record" option as a choice in
      the 'Schema Access Strategy' of writers.

      Attachments

        Issue Links

          Activity

            People

              markap14 Mark Payne
              ijokarumawak Koji Kawamura
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: