[AVRO-2817] Avro file generated using avro-1.8.2 is not readable from avro-1.9.2 for certian type of schemas

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.9.2
• Fix Version/s: 1.10.1
• Component/s: java
• Assignee: Ryan Skraba
• Reporter: Manoj Kumar Jha
• Created: 24/Apr/20 15:44
• Updated: 20/Feb/21 09:40
• Resolved: 14/Oct/20 11:09

Description

Hello,

We observed   avro file generated using avro-1.8.2  is not readable while using avro-1.9.2   if schema contains a field "default:null".   Please see below.

 

--------------------------------------------------------------------------

[mxj142:Jars]$ ls -ltr
total 168520
rwx-----@ 1 mxj142 staff 34798932 Jan 16 14:45 avro-tools-1.8.2.jar
-rwxr-xr-x@ 1 mxj142 staff 51303364 Mar 6 17:43 avro-tools-1.9.2.jar
rw-rr- 1 mxj142 staff 715 Apr 24 11:19 records.avro
[mxj142:Jars]$ java -jar avro-tools-1.8.2.jar tojson records.avro
log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

{"enrichmentHeader":\{"correlationId":"1","sourceId":"09b5d770-7373-462d-9517-8e5ce957793f"}

,"cx":{"dateFirst":"20200419","dateLast":"20200419"}}

{"enrichmentHeader":\{"correlationId":"2","sourceId":"0285dc49-cc5c-4717-8ea2-45e50de2f65f"}

,"cx":{"dateFirst":"20200420","dateLast":"20200420"}}
[mxj142:Jars]$
[mxj142:Jars]$
[mxj142:Jars]$
[mxj142:Jars]$
[mxj142:Jars]$
[mxj142:Jars]$ java -jar avro-tools-1.9.2.jar tojson records.avro
20/04/24 11:20:51 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" org.apache.avro.AvroTypeException: Invalid default for field enrichmentHeader: null not a {"type":"record","name":"EnrichmentHeader","namespace":"com.mxj142.commons.domain","fields":[{"name":"correlationId","type":{"type":"string","avro.java.string":"String"}},\{"name":"sourceId","type":{"type":"string","avro.java.string":"String"}}]}
at org.apache.avro.Schema.validateDefault(Schema.java:1540)
at org.apache.avro.Schema.access$500(Schema.java:87)
at org.apache.avro.Schema$Field.<init>(Schema.java:521)
at org.apache.avro.Schema.parse(Schema.java:1647)
at org.apache.avro.Schema$Parser.parse(Schema.java:1394)
at org.apache.avro.Schema$Parser.parse(Schema.java:1382)
at org.apache.avro.file.DataFileStream.initialize(DataFileStream.java:130)
at org.apache.avro.file.DataFileStream.<init>(DataFileStream.java:90)
at org.apache.avro.tool.DataFileReadTool.run(DataFileReadTool.java:93)
at org.apache.avro.tool.Main.run(Main.java:66)
at org.apache.avro.tool.Main.main(Main.java:55)
[mxj142:Jars]$

--------------------------------------------------------------------------

 

The file "records.avro"  is generated using avro-1.8.2 .  It contains two  records  and using "avro-tools-1.8.2.jar"  one can read it.  If   we use   "avro-tools-1.9.2.jar", then "records.avro"  is not readable  (as shown above).  The file "records.avro"  is also  attached for your convenience. 

 

Let me know if you need any more information.

 

Thanks,

Manoj

 

 

Issue Links

• is related to: SPARK-27733 Upgrade to Avro 1.10.1
• relates to: AVRO-2366 setValidateDefaults=false does not seem to work
• relates to: HIVE-24797 Disable validate default values when parsing Avro schemas
• links to: GitHub Pull Request #966