[IMPALA-9453] S3 build failed with many strange symptoms

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Won't Fix
• Affects Version/s: Impala 3.4.0
• Fix Version/s: Not Applicable
• Labels: broken-build crash
• Target Version: Impala 4.0.0
• Epic Color: ghx-label-9
• Assignee: Unassigned
• Reporter: Tim Armstrong
• Created: 02/Mar/20 23:33
• Updated: 04/Dec/20 19:32
• Resolved: 04/Dec/20 19:32

Description

There were a lot of incorrect results:

uery_test/test_mt_dop.py:49: in test_mt_dop     self.run_test_case('QueryTest/mt-dop', new_vector) common/impala_test_suite.py:690: in run_test_case     self.__verify_results_and_errors(vector, test_section, result, use_db) common/impala_test_suite.py:523: in __verify_results_and_errors     replace_filenames_with_placeholder) common/test_result_verifier.py:456: in verify_raw_results     VERIFIER_MAP[verifier](expected, actual) common/test_result_verifier.py:278: in verify_query_result_is_equal     assert expected_results == actual_results E   assert Comparing QueryTestResults (expected vs actual): E     7300 != 6990

Stacktrace

query_test/test_mt_dop.py:49: in test_mt_dop
    self.run_test_case('QueryTest/mt-dop', new_vector)
common/impala_test_suite.py:690: in run_test_case
    self.__verify_results_and_errors(vector, test_section, result, use_db)
common/impala_test_suite.py:523: in __verify_results_and_errors
    replace_filenames_with_placeholder)
common/test_result_verifier.py:456: in verify_raw_results
    VERIFIER_MAP[verifier](expected, actual)
common/test_result_verifier.py:278: in verify_query_result_is_equal
    assert expected_results == actual_results
E   assert Comparing QueryTestResults (expected vs actual):
E     7300 != 6990

Standard Error

ERROR:test_configuration:Comparing QueryTestResults (expected vs actual):
7300 != 6990

The impalads eventually crashed:

F0302 00:50:55.607841   483 parquet-page-reader.cc:67] e24eb0839fa75423:8ac0bf7300000002] Check failed: col_end < file_desc.file_length (7010 vs. 7010) 
*** Check failure stack trace: ***
    @          0x4f7277c  google::LogMessage::Fail()
    @          0x4f74021  google::LogMessage::SendToLog()
    @          0x4f72156  google::LogMessage::Flush()
    @          0x4f7571d  google::LogMessageFatal::~LogMessageFatal()
    @          0x2e3a520  impala::ParquetPageReader::InitColumnChunk()
    @          0x2e37dee  impala::ParquetColumnChunkReader::InitColumnChunk()
    @          0x2cd8000  impala::BaseScalarColumnReader::Reset()
    @          0x2c91239  impala::HdfsParquetScanner::InitScalarColumns()
    @          0x2c8775a  impala::HdfsParquetScanner::NextRowGroup()
    @          0x2c85c2a  impala::HdfsParquetScanner::GetNextInternal()
    @          0x2c8403e  impala::HdfsParquetScanner::ProcessSplit()
    @          0x28b826b  impala::HdfsScanNode::ProcessSplit()
    @          0x28b7440  impala::HdfsScanNode::ScannerThread()
    @          0x28b679d  _ZZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS_18ThreadResourcePoolEENKUlvE_clEv
    @          0x28b8d91  _ZN5boost6detail8function26void_function_obj_invoker0IZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS3_18ThreadResourcePoolEEUlvE_vE6invokeERNS1_15function_bufferE
    @          0x20aa1e9  boost::function0<>::operator()()
    @          0x266a84a  impala::Thread::SuperviseThread()
    @          0x2672ace  boost::_bi::list5<>::operator()<>()
    @          0x26729f2  boost::_bi::bind_t<>::operator()()
    @          0x26729b5  boost::detail::thread_data<>::run()
    @          0x3e98b19  thread_proxy
    @     0x7f8c2ccefe24  start_thread
    @     0x7f8c2985b34c  __clone

F0302 00:50:41.466794 32643 parquet-page-reader.cc:67] dd48a46583bea9c8:e3be641000000002] Check failed: col_end < file_desc.file_length (7010 vs. 7010) 
*** Check failure stack trace: ***
    @          0x4f7277c  google::LogMessage::Fail()
    @          0x4f74021  google::LogMessage::SendToLog()
    @          0x4f72156  google::LogMessage::Flush()
    @          0x4f7571d  google::LogMessageFatal::~LogMessageFatal()
    @          0x2e3a520  impala::ParquetPageReader::InitColumnChunk()
    @          0x2e37dee  impala::ParquetColumnChunkReader::InitColumnChunk()
    @          0x2cd8000  impala::BaseScalarColumnReader::Reset()
    @          0x2c91239  impala::HdfsParquetScanner::InitScalarColumns()
    @          0x2c8775a  impala::HdfsParquetScanner::NextRowGroup()
    @          0x2c85c2a  impala::HdfsParquetScanner::GetNextInternal()
    @          0x2c8403e  impala::HdfsParquetScanner::ProcessSplit()
    @          0x28b826b  impala::HdfsScanNode::ProcessSplit()
    @          0x28b7440  impala::HdfsScanNode::ScannerThread()
    @          0x28b679d  _ZZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS_18ThreadResourcePoolEENKUlvE_clEv
    @          0x28b8d91  _ZN5boost6detail8function26void_function_obj_invoker0IZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS3_18ThreadResourcePoolEEUlvE_vE6invokeERNS1_15function_bufferE
    @          0x20aa1e9  boost::function0<>::operator()()
    @          0x266a84a  impala::Thread::SuperviseThread()
    @          0x2672ace  boost::_bi::list5<>::operator()<>()
    @          0x26729f2  boost::_bi::bind_t<>::operator()()
    @          0x26729b5  boost::detail::thread_data<>::run()
    @          0x3e98b19  thread_proxy
    @     0x7f16a5766e24  start_thread
    @     0x7f16a22d234c  __clone
Wrote minidump to /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/logs/ee_tests/minidumps/impalad/49436f97-da0e-47c9-76b372a5-6b3fb146.dmp

If it makes a differnence, this was using CDP, so that version of the S3 connector.

Issue Links

• relates to: IMPALA-9759 Revisit integration of snapshot dataload with s3guard