Uploaded image for project: 'Parquet'
  1. Parquet
  2. PARQUET-2039

AvroReadSupport.setRequestedProjection in 1.11.1+ not backwards compatible with MAPS

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 1.11.0, 1.11.1
    • Fix Version/s: None
    • Component/s: parquet-avro, parquet-mr
    • Labels:
      None

      Description

       

      using AvroReadSupport.setRequestedProjection in 1.11.1 reading a 1.10.0 generated parquet file sets MAP<STRING,STRING> fields to null (and vice versa 1.10.0 reader against a 1.11.1 generated file) 

      • Not using a projected schema works, the map fields are converted correctly.
        In my case. Parquet file is generated by hive 
        CREATE TABLE parquetmaptest (
         a string,
         b MAP<string,string>
        )
        STORED AS PARQUET 
        tblproperties(
         "parquet.compression"="SNAPPY"
        );
        insert into parquetmaptest select "a",map("k1","v1","k2","v2");

      Using parquet-avro 1.11.1 (and appropriate dependencies) result in field "b" being null. 

      data:null
      row:{"a": "a", "b": null}

      Using parquet-avro 1.11.0 (and appropriate dependencies) result in field "b" being  the right map value.  

      data:{k1=v1, k2=v2}
      row:{"a": "a", "b": {"k1": "v1", "k2": "v2"}}

       

        Attachments

        1. PQMapTest.java
          4 kB
          Bil Bingham

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              bilbingham Bil Bingham
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: