Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-9249

HdfsOrcScanner crash on resolving file schema

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Backend

      Description

       Encounter a crash when running test_scanners_fuzz test on orc format:

      F1214 12:35:18.365988  4945 orc-metadata-utils.cc:29] c749de778e550dcb:50f6705f00000001] Check failed: root.getKind() == orc::TypeKind::STRUCT (0 vs. 12)
      *** Check failure stack trace: ***
          @          0x4d864fc  google::LogMessage::Fail()
          @          0x4d87da1  google::LogMessage::SendToLog()
          @          0x4d85ed6  google::LogMessage::Flush()
          @          0x4d8949d  google::LogMessageFatal::~LogMessageFatal()
          @          0x2622c73  impala::OrcMetadataUtils::BuildSchemaPaths()
          @          0x25e49ff  impala::HdfsOrcScanner::Open()
          @          0x2570e06  impala::HdfsScanNodeBase::CreateAndOpenScannerHelper()
          @          0x256418f  impala::HdfsScanNode::ProcessSplit()
          @          0x25636f4  impala::HdfsScanNode::ScannerThread()
          @          0x2562a7d  _ZZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS_18ThreadResourcePoolEENKUlvE_clEv
          @          0x2565045  _ZN5boost6detail8function26void_function_obj_invoker0IZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS3_18ThreadResourcePoolEEUlvE_vE6invokeERNS1_15function_bufferE
          @          0x1f1e469  boost::function0<>::operator()()
          @          0x24aaec8  impala::Thread::SuperviseThread()
          @          0x24b324c  boost::_bi::list5<>::operator()<>()
          @          0x24b3170  boost::_bi::bind_t<>::operator()()
          @          0x24b3133  boost::detail::thread_data<>::run()
          @          0x3caf6e9  thread_proxy
          @     0x7f4b5be7a6b9  start_thread
          @     0x7f4b5863641c  clone

      Code snipper:

      void OrcMetadataUtils::BuildSchemaPaths(const orc::Type& root, int num_partition_keys,
          vector<SchemaPath>* paths) {
        SchemaPath path;
        paths->push_back(path);
        DCHECK_EQ(root.getKind(), orc::TypeKind::STRUCT); <-- It could be wrong for corrupt files
        int num_columns = root.getSubtypeCount();
        for (int i = 0; i < num_columns; ++i) {
          path.push_back(i + num_partition_keys);
          BuildSchemaPath(*root.getSubtype(i), &path, paths);
          path.pop_back();
        }
      }
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                csringhofer Csaba Ringhofer
                Reporter:
                stigahuang Quanlong Huang
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: