Uploaded image for project: 'Parquet'
  1. Parquet
  2. PARQUET-215

Parquet Thrift should discard records with unrecognized union members

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.6.0
    • Component/s: parquet-mr
    • Labels:
      None

      Description

      When writing parquet-thrift files, when a thrift record with an unknown union member is encountered, it should be considered a bad record and discarded. Currently, because unions are treated as structs with one optional field per union member, parquet-thrift happily writes the empty struct, but then crashes in the read path when trying to read this record.

      We should discard these records in the write path, just as we discard other unparseable records.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              alexlevenson Alex Levenson
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: