[PARQUET-2039] AvroReadSupport.setRequestedProjection in 1.11.1+ not backwards compatible with MAPS

• Type: Bug
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 1.11.0, 1.11.1
• Component/s: parquet-avro, parquet-mr
• Assignee: Unassigned
• Reporter: Bil Bingham
• Created: 28/Apr/21 21:03
• Updated: 29/Apr/21 14:34

Description

 

using AvroReadSupport.setRequestedProjection in 1.11.1 reading a 1.10.0 generated parquet file sets MAP<STRING,STRING> fields to null (and vice versa 1.10.0 reader against a 1.11.1 generated file) 

• Not using a projected schema works, the map fields are converted correctly.
  In my case. Parquet file is generated by hive 

  CREATE TABLE parquetmaptest (
   a string,
   b MAP<string,string>
  )
  STORED AS PARQUET 
  tblproperties(
   "parquet.compression"="SNAPPY"
  );
  insert into parquetmaptest select "a",map("k1","v1","k2","v2");

Using parquet-avro 1.11.1 (and appropriate dependencies) result in field "b" being null. 

data:null
row:{"a": "a", "b": null}

Using parquet-avro 1.11.0 (and appropriate dependencies) result in field "b" being  the right map value.  

data:{k1=v1, k2=v2}
row:{"a": "a", "b": {"k1": "v1", "k2": "v2"}}