Uploaded image for project: 'Parquet'
  1. Parquet
  2. PARQUET-180

Parquet-thrift compile issue with 0.9.2.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.6.0
    • Component/s: None
    • Labels:
      None

      Description

      Thrift 0.9.2 removed setReadLength. This causes parquet-thrift to fail because it is called for TBinaryProtocol. The reason we use it is defensive: a size is read from the data and then that many bytes are read, so using this method sets a maximum and causes an exception rather than a strange failure later on. The code also has a comment that says it is okay when it can't be used.

          /* Reduce the chance of OOM when data is corrupted. When readBinary is called on TBinaryProtocol, it reads the length of the binary first,
           so if the data is corrupted, it could read a big integer as the length of the binary and therefore causes OOM to happen.
           Currently this fix only applies to TBinaryProtocol which has the setReadLength defined.
            */
          if (protocol instanceof TBinaryProtocol) {
            ((TBinaryProtocol)protocol).setReadLength(record.getLength());
          }
      

      I think the fix is to remove the section above.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              rdblue Ryan Blue
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: