Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
Impala 1.1
-
None
-
None
Description
Occasionally fail in BE due to error "Could not find partition with id: X". This is something we had seen in the past and I added some extra debug logging, so now there is a bit more context. It the BE is expecting a partition ID of 2686 but the table only contains partition id 2687 and -1 (ImpalaInternalServiceConstants.DEFAULT_PARTITION_ID).
Log info:
I0806 17:39:51.018662 1909 plan-fragment-executor.cc:221] Open(): instance_id=4b4cdd244e8ca854:6ff5b28bbdbd7e94 I0806 17:39:51.184165 1875 data-stream-mgr.cc:274] DeregisterRecvr(): fragment_instance_id=4b4cdd244e8ca854:6ff5b28bbdbd7e8e, node=4 I0806 17:39:51.184231 1875 data-stream-mgr.cc:170] cancelled stream: fragment_instance_id_=4b4cdd244e8ca854:6ff5b28bbdbd7e8e node_id=4 I0806 17:39:51.190831 1875 data-stream-mgr.cc:274] DeregisterRecvr(): fragment_instance_id=4b4cdd244e8ca854:6ff5b28bbdbd7e8e, node=5 I0806 17:39:51.199070 1875 data-stream-mgr.cc:170] cancelled stream: fragment_instance_id_=4b4cdd244e8ca854:6ff5b28bbdbd7e8e node_id=5 I0806 17:39:51.916009 2015 impala-server.cc:1192] ExecPlanFragment() instance_id=84cd3f3ebe1c76d:f0d28bc0fc94088c coord=a2123.halxg.cloudera.com:22000 backend#=1 I0806 17:39:51.916116 2015 plan-fragment-executor.cc:76] Prepare(): query_id=84cd3f3ebe1c76d:f0d28bc0fc940889 instance_id=84cd3f3ebe1c76d:f0d28bc0fc94088c I0806 17:39:51.940984 2015 plan-fragment-executor.cc:124] descriptor table for fragment=84cd3f3ebe1c76d:f0d28bc0fc94088c tuples: Tuple(id=0 size=32 slots=[Slot(id=1 type=STRING col=1 offset=16 null=(offset=0 mask=4)), Slot(id=2 type=BIGINT col=0 offset=8 null=(offset=0 mask=2)), Slot(id=3 type=INT col=2 offset=4 null=(offset=0 mask=1))]) Tuple(id=1 size=24 slots=[Slot(id=0 type=STRING col=1 offset=8 null=(offset=0 mask=1))]) I0806 17:39:52.005525 1955 plan-fragment-executor.cc:221] Open(): instance_id=84cd3f3ebe1c76d:f0d28bc0fc94088c I0806 17:39:52.005878 1955 hdfs-scan-node.cc:517] HdfsTable(#cols=3 #clustering_cols=0 hdfs_base_dir='hdfs://localhost:20500/test-warehouse/dimtbl_parquet' col_names=[id:name:zip] partitions=[ (id: -1, partition: file_format=5' line_delim=' ' field_delim='^A' coll_delim='^A' escape_char='^@')), (id: 2687, partition: file_format=5' line_delim=' ' field_delim='^A' coll_delim='^A' escape_char='^@'))] null_partition_key_value='__HIVE_DEFAULT_PARTITION__' null_column_value='\N' I0806 17:39:52.035392 1955 status.cc:44] Could not find partition with id: 2686 @ 0xdd4eab impala::Status::Status() @ 0xedcb18 impala::HdfsScanNode::Open() @ 0xd6725e impala::PlanFragmentExecutor::OpenInternal() @ 0xd670d3 impala::PlanFragmentExecutor::Open() @ 0xb409d1 impala::ImpalaServer::FragmentExecState::Exec() @ 0xb4fa16 impala::ImpalaServer::RunExecPlanFragment() @ 0xbadb36 boost::_mfi::mf1<>::operator()() @ 0xbad5e7 boost::_bi::list2<>::operator()<>() @ 0xbad443 boost::_bi::bind_t<>::operator()() @ 0xbac85c boost::detail::thread_data<>::run() @ 0xf93ff4 thread_proxy @ 0x3a67407851 (unknown) @ 0x3a670e890d (unknown) I0806 17:39:52.809054 16254 impala-server.cc:1192] ExecPlanFragment() instance_id=14bedd4ee8dce1b:6e646e4cac0a08be coord=a2123.halxg.cloudera.com:22000 backend#=1 I0806 17:39:52.809231 16254 plan-fragment-executor.cc:76] Prepare(): query_id=14bedd4ee8dce1b:6e646e4cac0a08bb instance_id=14bedd4ee8dce1b:6e646e4cac0a08be I0806 17:39:52.831631 16254 plan-fragment-executor.cc:124] descriptor table for fragment=14bedd4ee8dce1b:6e646e4cac0a08be tuples:
Test Failure
17:33:57 [gw2] FAILED query_test/test_join_queries.py:29: TestJoinQueries.test_outer_joins[batch_size: 0 | exec_option: {'disable_codegen': False, 'abort_on_error': 1, 'batch_size': 0, 'num_nodes': 0} | table_format: seq/snap/block]
17:33:57
17:33:57 =================================== FAILURES ===================================
17:33:57 TestJoinQueries.test_outer_joins[batch_size: 0 | exec_option: {'disable_codegen': False, 'abort_on_error': 1, 'batch_size': 0, 'num_nodes': 0} | table_format: seq/snap/block]
17:33:57 [gw2] linux2 -- Python 2.6.6 /usr/bin/python
17:33:57 query_test/test_join_queries.py:32: in test_outer_joins
17:33:57 > self.run_test_case('QueryTest/outer-joins', new_vector)
17:33:57 common/impala_test_suite.py:140: in run_test_case
17:33:57 > result = self.execute_query_expect_success(IMPALAD, query, exec_options)
17:33:57 common/impala_test_suite.py:208: in wrapper
17:33:57 > return function(*args, **kwargs)
17:33:57 common/impala_test_suite.py:214: in execute_query_expect_success
17:33:57 > result = self.__execute_query(impalad, query, query_exec_options)
17:33:57 common/impala_test_suite.py:271: in __execute_query
17:33:57 > return self.client.execute(query)
17:33:57 beeswax/impala_beeswax.py:147: in execute
17:33:57 > handle = self.__execute_query(query_string.strip())
17:33:57 beeswax/impala_beeswax.py:177: in __execute_query
17:33:57 > self.wait_for_completion(handle)
17:33:57 beeswax/impala_beeswax.py:193: in wait_for_completion
17:33:57 > raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
17:33:57 E ImpalaBeeswaxException: <class 'tests.beeswax.impala_beeswax.ImpalaBeeswaxException'>:
17:33:57 E Query aborted:
17:33:57 E Backend 6:Could not find partition with id: 2684
17:33:57 E Backend 7:Could not find partition with id: 2685
17:33:57 E Backend 8:Could not find partition with id: 2686
17:33:57 ------------------------------- Captured stderr --------------------------------