Uploaded image for project: 'Apache Avro'
  1. Apache Avro
  2. AVRO-672

Convert JSON Text Input to Avro Tool

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsAdd voteVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Patch Available
    • Major
    • Resolution: Unresolved
    • None
    • None
    • java
    • None

    Description

      The attached patch allows reading a JSON-formatted text file in, converting to a conforming Avro text file, emitting one record per line, e.g., it can read this input file:

      {"intval":12} {"intval":-73,"strval":"hello, there!!"}

      with this schema:
      { "type":"record", "name":"TestRecord", "fields": [

      {"name":"intval","type":"int"}

      ,

      {"name":"strval","type":["string", "null"]}

      ]}

      returning valid Avro. This is different than the DataFileWriteTool, which would read in the following internal encoding:

      {"intval":12,"strval":null}

      {"intval":-73,"strval":{"string":"hello, there!!"}}

      In general, the internal encodings used by Avro aren't natural when reading in JSON text that appears in the wild. Likewise, this utility allows changing invalid Avro identifier characters into an underscore, again to tolerate JSON that wasn't designed to be readable by Avro.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned Assign to me
            rbodkin Ron Bodkin

            Dates

              Created:
              Updated:

              Slack

                Issue deployment