Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-7080

DCHECK hit in DelimitedTextParser

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: Not Applicable
    • Component/s: None
    • Labels:
    • Epic Color:
      ghx-label-9

      Description

      Seen in a 2.x exhaustive run:

      #0  0x00007eff94d2e8e5 in raise () from /lib64/libc.so.6
      #1  0x00007eff94d300c5 in abort () from /lib64/libc.so.6
      #2  0x0000000003ffc3a4 in google::DumpStackTraceAndExit() ()
      #3  0x0000000003ff2e1d in google::LogMessage::Fail() ()
      #4  0x0000000003ff46c2 in google::LogMessage::SendToLog() ()
      #5  0x0000000003ff27f7 in google::LogMessage::Flush() ()
      #6  0x0000000003ff5dbe in google::LogMessageFatal::~LogMessageFatal() ()
      #7  0x0000000002afcbde in impala::DelimitedTextParser<false>::ParseFieldLocations (this=0x17bc5e80, max_tuples=1, remaining_len=-60, byte_buffer_ptr=0x7efed9538ab0, row_end_locations=0x7efed9538aa0, field_locations=0x152c4000, num_tuples=0x7efed9538aac, num_fields=0x7efed9538aa8, next_column_start=0x7efed9538ab8) at /data/jenkins/workspace/impala-asf-2.x-exhaustive/repos/Impala/be/src/exec/delimited-text-parser.cc:205
      #8  0x0000000001d47d61 in impala::HdfsSequenceScanner::ProcessRange (this=0x17dd0ac0, row_batch=0xde4b9a0) at /data/jenkins/workspace/impala-asf-2.x-exhaustive/repos/Impala/be/src/exec/hdfs-sequence-scanner.cc:352
      #9  0x0000000002aebe3e in impala::BaseSequenceScanner::GetNextInternal (this=0x17dd0ac0, row_batch=0xde4b9a0) at /data/jenkins/workspace/impala-asf-2.x-exhaustive/repos/Impala/be/src/exec/base-sequence-scanner.cc:181
      #10 0x0000000001d1e70a in impala::HdfsScanner::ProcessSplit (this=0x17dd0ac0) at /data/jenkins/workspace/impala-asf-2.x-exhaustive/repos/Impala/be/src/exec/hdfs-scanner.cc:134
      #11 0x0000000001cf5834 in impala::HdfsScanNode::ProcessSplit (this=0xd6b3800, filter_ctxs=..., expr_results_pool=0x7efed9539490, scan_range=0xd8d9500, scanner_thread_reservation=0x7efed9539408) at /data/jenkins/workspace/impala-asf-2.x-exhaustive/repos/Impala/be/src/exec/hdfs-scan-node.cc:453
      #12 0x0000000001cf4bd5 in impala::HdfsScanNode::ScannerThread (this=0xd6b3800, first_thread=true, scanner_thread_reservation=32768) at /data/jenkins/workspace/impala-asf-2.x-exhaustive/repos/Impala/be/src/exec/hdfs-scan-node.cc:360
      #13 0x0000000001cf4048 in impala::HdfsScanNode::<lambda()>::operator()(void) const (__closure=0x7efed9539bc8) at /data/jenkins/workspace/impala-asf-2.x-exhaustive/repos/Impala/be/src/exec/hdfs-scan-node.cc:292
      #14 0x0000000001cf60a4 in boost::detail::function::void_function_obj_invoker0<impala::HdfsScanNode::ThreadTokenAvailableCb(impala::ThreadResourcePool*)::<lambda()>, void>::invoke(boost::detail::function::function_buffer &) (function_obj_ptr=...) at /data/jenkins/workspace/impala-asf-2.x-exhaustive/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:153
      #15 0x0000000001929fca in boost::function0<void>::operator() (this=0x7efed9539bc0) at /data/jenkins/workspace/impala-asf-2.x-exhaustive/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:767
      #16 0x0000000001c48275 in impala::Thread::SuperviseThread (name=..., category=..., functor=..., parent_thread_info=0x7efed8b38870, thread_started=0x7efed8b376d0) at /data/jenkins/workspace/impala-asf-2.x-exhaustive/repos/Impala/be/src/util/thread.cc:356
      #17 0x0000000001c5074b in boost::_bi::list5<boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<boost::function<void()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long int>*> >::operator()<void (*)(const std::basic_string<char>&, const std::basic_string<char>&, boost::function<void()>, const impala::ThreadDebugInfo*, impala::Promise<long int>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(const std::basic_string<char, std::char_traits<char>, std::allocator<char> > &, const std::basic_string<char, std::char_traits<char>, std::allocator<char> > &, boost::function<void()>, const impala::ThreadDebugInfo *, impala::Promise<long> *), boost::_bi::list0 &, int) (this=0xd37a65c0, f=@0xd37a65b8, a=...) at /data/jenkins/workspace/impala-asf-2.x-exhaustive/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind.hpp:525
      #18 0x0000000001c5066f in boost::_bi::bind_t<void, void (*)(const std::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const std::basic_string<char, std::char_traits<char>, std::allocator<char> >&, boost::function<void()>, const impala::ThreadDebugInfo*, impala::Promise<long int>*), boost::_bi::list5<boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<boost::function<void()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long int>*> > >::operator()(void) (this=0xd37a65b8) at /data/jenkins/workspace/impala-asf-2.x-exhaustive/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind_template.hpp:20
      #19 0x0000000001c50632 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(const std::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const std::basic_string<char, std::char_traits<char>, std::allocator<char> >&, boost::function<void()>, const impala::ThreadDebugInfo*, impala::Promise<long int>*), boost::_bi::list5<boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<boost::function<void()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long int>*> > > >::run(void) (this=0xd37a6400) at /data/jenkins/workspace/impala-asf-2.x-exhaustive/Impala-Toolchain/boost-1.57.0-p3/include/boost/thread/detail/thread.hpp:116
      #20 0x0000000002f1fd2a in thread_proxy ()
      #21 0x00007eff95096851 in start_thread () from /lib64/libpthread.so.0
      #22 0x00007eff94de494d in clone () from /lib64/libc.so.6
      

      excerpt from impalad.INFO

      I0525 12:16:04.500361 21484 initial-reservations.cc:60] Successfully claimed initial reservations (6.02 MB) for query e4caae1b4cd8623:da0da86400000000
      I0525 12:16:04.511389 27250 query-state.cc:300] StartFInstances(): query_id=e4caae1b4cd8623:da0da86400000000 #instances=1
      I0525 12:16:04.511759 27250 query-state.cc:313] descriptor table for query=e4caae1b4cd8623:da0da86400000000
      tuples:
      Tuple(id=7 size=5 slots=[Slot(id=7 type=INT col_path=[] offset=0 null=(offset=4 mask=1) slot_idx=0 field_idx=-1)] tuple_path=[])
      Tuple(id=6 size=5 slots=[Slot(id=6 type=INT col_path=[] offset=0 null=(offset=4 mask=1) slot_idx=0 field_idx=-1)] tuple_path=[])
      Tuple(id=5 size=9 slots=[Slot(id=5 type=BIGINT col_path=[] offset=0 null=(offset=8 mask=1) slot_idx=0 field_idx=-1)] tuple_path=[])
      Tuple(id=4 size=8 slots=[Slot(id=4 type=BIGINT col_path=[] offset=0 null=(offset=0 mask=0) slot_idx=0 field_idx=-1)] tuple_path=[])
      Tuple(id=3 size=9 slots=[Slot(id=3 type=BIGINT col_path=[] offset=0 null=(offset=8 mask=1) slot_idx=0 field_idx=-1)] tuple_path=[])
      Tuple(id=0 size=5 slots=[Slot(id=0 type=INT col_path=[3] offset=0 null=(offset=4 mask=1) slot_idx=0 field_idx=-1)] tuple_path=[])
      I0525 12:16:04.519618 27253 query-state.cc:395] Executing instance. instance_id=e4caae1b4cd8623:da0da86400000001 fragment_idx=1 per_fragment_instance_idx=0 coord_state_idx=2 #in-flight=5
      I0525 12:16:04.519969 27253 hdfs-scan-node.cc:147] Max row batch queue size for scan node '0' in fragment instance 'e4caae1b4cd8623:da0da86400000001': 60
      I0525 12:16:04.540093 27250 query-exec-mgr.cc:155] ReleaseQueryState(): query_id=e4caae1b4cd8623:da0da86400000000 refcnt=2
      I0525 12:16:04.684156 27253 query-state.cc:403] Instance completed. instance_id=e4caae1b4cd8623:da0da86400000001 #in-flight=4 status=OK
      I0525 12:16:04.684221 27253 query-exec-mgr.cc:155] ReleaseQueryState(): query_id=e4caae1b4cd8623:da0da86400000000 refcnt=1
      I0525 12:16:04.951737 27189 status.cc:53] Snappy: GetUncompressedLength failed
          @          0x176ae02  impala::Status::Status()
          @          0x1bbcb7b  SnappyBlockDecompress()
          @          0x1bbce02  impala::SnappyBlockDecompressor::ProcessBlock()
          @          0x1d4ad34  impala::HdfsSequenceScanner::GetRecord()
          @          0x1d47bc0  impala::HdfsSequenceScanner::ProcessRange()
          @          0x2aebe3e  impala::BaseSequenceScanner::GetNextInternal()
          @          0x1d1e70a  impala::HdfsScanner::ProcessSplit()
          @          0x1cf5834  impala::HdfsScanNode::ProcessSplit()
          @          0x1cf4bd5  impala::HdfsScanNode::ScannerThread()
          @          0x1cf4048  _ZZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS_18ThreadResourcePoolEENKUlvE_clEv
          @          0x1cf60a4  _ZN5boost6detail8function26void_function_obj_invoker0IZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS3_18ThreadResourcePoolEEUlvE_vE6invokeERNS1_15function_bufferE
          @          0x1929fca  boost::function0<>::operator()()
          @          0x1c48275  impala::Thread::SuperviseThread()
          @          0x1c5074b  boost::_bi::list5<>::operator()<>()
          @          0x1c5066f  boost::_bi::bind_t<>::operator()()
          @          0x1c50632  boost::detail::thread_data<>::run()
          @          0x2f1fd2a  thread_proxy
          @     0x7eff95096851  (/lib64/libpthread-2.12.so.#prelink#.nB8q5z (deleted)+0x7850)
          @     0x7eff94de494d  (/lib64/libc-2.12.so (deleted)+0xe894c)
      I0525 12:16:04.951784 27189 runtime-state.cc:175] Error from query 5042c413dedf5fb3:1c1adc9b00000000: Problem parsing file hdfs://localhost:20500/test-warehouse/test_fuzz_alltypes_fc3d0635.db/alltypes/year=2009/month=3/000012_0 at 8781
      I0525 12:16:04.951802 27189 runtime-state.cc:175] Error from query 5042c413dedf5fb3:1c1adc9b00000000: Snappy: GetUncompressedLength failed
      F0525 12:16:04.959331 27189 delimited-text-parser.cc:205] Check failed: remaining_len == 0 (-60 vs. 0) 
      I0525 12:16:05.056919 21486 impala-internal-service.cc:44] ExecQueryFInstances(): query_id=344b4f95d6cb1235:3c1e97ad00000000
      I0525 12:16:05.056996 21486 query-exec-mgr.cc:46] StartQueryFInstances() query_id=344b4f95d6cb1235:3c1e97ad00000000 coord=ec2-m2-4xlarge-centos-6-4-0b75.vpc.cloudera.com:22000
      I0525 12:16:05.057194 21486 query-state.cc:178] Buffer pool limit for 344b4f95d6cb1235:3c1e97ad00000000: 10307921510
      I0525 12:16:05.057317 21486 initial-reservations.cc:60] Successfully claimed initial reservations (6.02 MB) for query 344b4f95d6cb1235:3c1e97ad00000000
      I0525 12:16:05.064730 27272 query-state.cc:300] StartFInstances(): query_id=344b4f95d6cb1235:3c1e97ad00000000 #instances=1
      I0525 12:16:05.065093 27272 query-state.cc:313] descriptor table for query=344b4f95d6cb1235:3c1e97ad00000000
      tuples:
      Tuple(id=7 size=5 slots=[Slot(id=7 type=INT col_path=[] offset=0 null=(offset=4 mask=1) slot_idx=0 field_idx=-1)] tuple_path=[])
      Tuple(id=6 size=5 slots=[Slot(id=6 type=INT col_path=[] offset=0 null=(offset=4 mask=1) slot_idx=0 field_idx=-1)] tuple_path=[])
      Tuple(id=5 size=9 slots=[Slot(id=5 type=BIGINT col_path=[] offset=0 null=(offset=8 mask=1) slot_idx=0 field_idx=-1)] tuple_path=[])
      Tuple(id=4 size=8 slots=[Slot(id=4 type=BIGINT col_path=[] offset=0 null=(offset=0 mask=0) slot_idx=0 field_idx=-1)] tuple_path=[])
      Tuple(id=3 size=9 slots=[Slot(id=3 type=BIGINT col_path=[] offset=0 null=(offset=8 mask=1) slot_idx=0 field_idx=-1)] tuple_path=[])
      Tuple(id=0 size=5 slots=[Slot(id=0 type=INT col_path=[3] offset=0 null=(offset=4 mask=1) slot_idx=0 field_idx=-1)] tuple_path=[])
      I0525 12:16:05.073236 27274 query-state.cc:395] Executing instance. instance_id=344b4f95d6cb1235:3c1e97ad00000001 fragment_idx=1 per_fragment_instance_idx=0 coord_state_idx=2 #in-flight=5
      I0525 12:16:05.073555 27274 hdfs-scan-node.cc:147] Max row batch queue size for scan node '0' in fragment instance '344b4f95d6cb1235:3c1e97ad00000001': 60
      I0525 12:16:05.096856 27272 query-exec-mgr.cc:155] ReleaseQueryState(): query_id=344b4f95d6cb1235:3c1e97ad00000000 refcnt=2
      I0525 12:16:05.269975 27274 query-state.cc:403] Instance completed. instance_id=344b4f95d6cb1235:3c1e97ad00000001 #in-flight=4 status=OK
      I0525 12:16:05.270043 27274 query-exec-mgr.cc:155] ReleaseQueryState(): query_id=344b4f95d6cb1235:3c1e97ad00000000 refcnt=1
      I0525 12:16:05.599388 11085 impala-internal-service.cc:44] ExecQueryFInstances(): query_id=5c414f45c0f7242b:ad08ead800000000
      I0525 12:16:05.599465 11085 query-exec-mgr.cc:46] StartQueryFInstances() query_id=5c414f45c0f7242b:ad08ead800000000 coord=ec2-m2-4xlarge-centos-6-4-0b75.vpc.cloudera.com:22000
      I0525 12:16:05.599735 11085 query-state.cc:178] Buffer pool limit for 5c414f45c0f7242b:ad08ead800000000: 10307921510
      I0525 12:16:05.599952 11085 initial-reservations.cc:60] Successfully claimed initial reservations (6.02 MB) for query 5c414f45c0f7242b:ad08ead800000000
      I0525 12:16:05.615347 27293 query-state.cc:300] StartFInstances(): query_id=5c414f45c0f7242b:ad08ead800000000 #instances=1
      I0525 12:16:05.615679 27293 query-state.cc:313] descriptor table for query=5c414f45c0f7242b:ad08ead800000000
      
      12:15:49 [gw0] PASSED query_test/test_queries.py::TestQueries::test_analytic_fns[exec_option: {'disable_codegen_rows_threshold': 0, 'disable_codegen': True, 'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold': '100', 'batch_size': 0, 'num_nodes': 0} | table_format: rc/bzip/block] 
      12:17:19 query_test/test_queries.py::TestQueries::test_analytic_fns[exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold': 0} | table_format: rc/bzip/block] 
      12:17:19 [gw3] FAILED query_test/test_spilling.py::TestSpillingDebugActionDimensions::test_spilling[exec_option: {'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@0.1', 'default_spillable_buffer_size': '256k'} | table_format: parquet/none] 
      12:17:19 query_test/test_spilling.py::TestSpillingDebugActionDimensions::test_spilling[exec_option: {'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@0.5', 'default_spillable_buffer_size': '256k'} | table_format: parquet/none] 
      12:17:19 [gw1] FAILED query_test/test_spilling.py::TestSpillingDebugActionDimensions::test_spilling_regression_exhaustive[exec_option: {'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@0.1', 'default_spillable_buffer_size': '256k'} | table_format: parquet/none] 
      12:17:19 query_test/test_spilling.py::TestSpillingDebugActionDimensions::test_spilling_regression_exhaustive[exec_option: {'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@0.5', 'default_spillable_buffer_size': '256k'} | table_format: parquet/none] 
      12:17:19 [gw0] FAILED query_test/test_queries.py::TestQueries::test_analytic_fns[exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold': 0} | table_format: rc/bzip/block] 
      12:20:21 query_test/test_queries.py::TestQueries::test_analytic_fns[exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': True, 'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold': 0} | table_format: rc/bzip/block] 
      12:20:21 [gw0] FAILED query_test/test_queries.py::TestQueries::test_analytic_fns[exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': True, 'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold': 0} | table_format: rc/bzip/block] 
      12:20:21 query_test/test_queries.py::TestQueries::test_analytic_fns[exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': True, 'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold': 0} | table_format: rc/snap/block] 
      12:20:21 [gw1] FAILED query_test/test_spilling.py::TestSpillingDebugActionDimensions::test_spilling_regression_exhaustive[exec_option: {'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@0.5', 'default_spillable_buffer_size': '256k'} | table_format: parquet/none] 
      12:20:21 query_test/test_spilling.py::TestSpillingDebugActionDimensions::test_spilling_regression_exhaustive[exec_option: {'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@0.9', 'default_spillable_buffer_size': '256k'} | table_format: parquet/none] 
      12:20:21 [gw3] FAILED query_test/test_spilling.py::TestSpillingDebugActionDimensions::test_spilling[exec_option: {'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@0.5', 'default_spillable_buffer_size': '256k'} | table_format: parquet/none] 
      12:25:52 query_test/test_spilling.py::TestSpillingDebugActionDimensions::test_spilling[exec_option: {'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@0.9', 'default_spillable_buffer_size': '256k'} | table_format: parquet/none] 
      12:25:52 [gw2] PASSED query_test/test_scanners_fuzz.py::TestScannersFuzzing::test_fuzz_alltypes[exec_option: {'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@1.0', 'abort_on_error': False, 'mem_limit': '512m', 'num_nodes': 0} | table_format: seq/snap/record] 
      12:25:52 [gw3] FAILED query_test/test_spilling.py::TestSpillingDebugActionDimensions::test_spilling[exec_option: {'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@0.9', 'default_spillable_buffer_size': '256k'} | table_format: parquet/none] 
      12:25:52 query_test/test_spilling.py::TestSpillingDebugActionDimensions::test_spilling_aggs[exec_option: {'debug_action': None, 'default_spillable_buffer_size': '256k'} | table_format: parquet/none] 
      12:25:52 [gw0] FAILED query_test/test_queries.py::TestQueries::test_analytic_fns[exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': True, 'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold': 0} | table_format: rc/snap/block] 
      12:25:58 [gw1] FAILED query_test/test_spilling.py::TestSpillingDebugActionDimensions::test_spilling_regression_exhaustive[exec_option: {'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@0.9', 'default_spillable_buffer_size': '256k'} | table_format: parquet/none] 
      

      Possibly related to IMPALA-6389

        Attachments

          Activity

            People

            • Assignee:
              lv Lars Volker
              Reporter:
              twmarshall Thomas Tauber-Marshall
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: