Uploaded image for project: 'ORC'
  1. ORC
  2. ORC-414

[C++] ORC files with malformed protobuf objects can crash a release build

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.5.3
    • Fix Version/s: 1.5.7, 1.6.1, 1.7.0
    • Component/s: C++
    • Labels:
      None

      Description

      ORC files can be produced by any external tools. Some corrupt files may contain malformed protobuf objects which can crash the process. The attachment is an example.

      In a debug build, protobuf will throw exceptions for this file:

      $ build/tools/src/orc-scan maleformed_protobuf.orc 
      [libprotobuf FATAL /mnt/volume1/impala-orc/orc/build/c++/libs/thirdparty/protobuf_ep-install/include/google/protobuf/repeated_field.h:1522] CHECK failed: (index) < (current_size_): 
      Caught exception in maleformed_protobuf.orc: CHECK failed: (index) < (current_size_):
      

      It hits a DCHECK which is eliminated in a release build.

      1518 template <typename TypeHandler>
      1519 inline const typename TypeHandler::Type&
      1520 RepeatedPtrFieldBase::Get(int index) const {
      1521   GOOGLE_DCHECK_GE(index, 0);
      1522   GOOGLE_DCHECK_LT(index, current_size_);
      1523   return *cast<TypeHandler>(rep_->elements[index]);
      1524 }
      

      In a release build, the process crash immediately, which means any system integrated with the orc-lib will crash when processing such kind of files.

      $ build/tools/src/orc-scan maleformed_protobuf.orc 
      Segmentation fault (core dumped)
      

      The stacktrace for this crash:

      #0  0x0000000000588c1e in orc::ReaderImpl::ReaderImpl(std::shared_ptr<orc::FileContents>, orc::ReaderOptions const&, unsigned long, unsigned long) ()
      #1  0x000000000058b1ee in orc::createReader(std::unique_ptr<orc::InputStream, std::default_delete<orc::InputStream> >, orc::ReaderOptions const&) ()
      #2  0x00000000005847c0 in scanFile (out=..., filename=0x7ffcf03a173d "maleformed_protobuf.orc", batchSize=batchSize@entry=1024) at /mnt/volume1/impala-orc/orc/tools/src/FileScan.cc:32
      #3  0x0000000000584150 in main (argc=<optimized out>, argv=<optimized out>) at /mnt/volume1/impala-orc/orc/tools/src/FileScan.cc:84
      

      We may need to introduce checksums to avoid this.

        Attachments

        1. malformed_protobuf.orc
          0.8 kB
          Quanlong Huang

          Issue Links

            Activity

              People

              • Assignee:
                stigahuang Quanlong Huang
                Reporter:
                stigahuang Quanlong Huang
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m