Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-1356

Create Avro Schema by inferring CSV and JSON data

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.4.1
    • Fix Version/s: 0.5.0
    • Component/s: Extensions
    • Labels:

      Description

      There are several situations where the ability to dynamically create an Avro schema is desired. Kite provides the ability to dynamically infer an Avro schema from both CSV and JSON data. Since NiFi already contains a kite bundle for converting csv and json to Avro this feature should be an easy add.

      I propose 2 new processors "InferAvroSchemaFromCSV" and "InferAvroSchemaFromJSON". These processors will reside inside of the existing "nifi-kite-bundle" and extend upon the already present third party libraries. Each processor will accept either CSV or JSON and produce as output the Avro Schema JSON and the original data that was presented to them. The processors will rely on kite to perform the actually inferring of the schema.

        Attachments

          Activity

            People

            • Assignee:
              jeremy.dyer Jeremy Dyer
              Reporter:
              jeremy.dyer Jeremy Dyer
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 24h
                24h
                Remaining:
                Remaining Estimate - 24h
                24h
                Logged:
                Time Spent - Not Specified
                Not Specified