[AVRO-672] Convert JSON Text Input to Avro Tool

• Type: New Feature
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Component/s: java
• Assignee: Unassigned
• Reporter: Ron Bodkin
• Created: 20/Sep/10 23:05
• Updated: 24/Jun/15 10:41

Description

The attached patch allows reading a JSON-formatted text file in, converting to a conforming Avro text file, emitting one record per line, e.g., it can read this input file:

{"intval":12} {"intval":-73,"strval":"hello, there!!"}

with this schema:
{ "type":"record", "name":"TestRecord", "fields": [

{"name":"intval","type":"int"}

,

{"name":"strval","type":["string", "null"]}

]}

returning valid Avro. This is different than the DataFileWriteTool, which would read in the following internal encoding:

{"intval":12,"strval":null}

{"intval":-73,"strval":{"string":"hello, there!!"}}

In general, the internal encodings used by Avro aren't natural when reading in JSON text that appears in the wild. Likewise, this utility allows changing invalid Avro identifier characters into an underscore, again to tolerate JSON that wasn't designed to be readable by Avro.