Details
-
Bug
-
Status: Resolved
-
Critical
-
Resolution: Fixed
-
Impala 0.7
-
None
-
None
Description
There are a few problems when attempting to insert into tables with timestamp partition keys.
1) For the positive case, even though an insert appears to work properly, attempts to reload the table metadata fail due to error "DATE/DATETIME/TIMESTAMP literals not supported"
2) The timestamp format is not checked so if the user enters an invalid timestamp value Impala uses an empty string as the value which causes problems when updating the metastore "incomplete partition name - missing <col_name>"
3) There are some DCHECKS that fail when attempting to scan these tables.
[localhost:21000] > create table test (i int) partitioned by (t timestamp); Query: create table test (i int) partitioned by (t timestamp) [localhost:21000] > insert into test partition(t='2009-01-01 00:00:00') select 1 from functional.alltypes lim limit 1; Query: insert into test partition(t='2009-01-01 00:00:00') select 1 from functional.alltypes lim limit 1 Inserted 1 rows in 0.81s [localhost:21000] > select * from test; Query: select * from test ERROR: AnalysisException: Failed to load metadata for table: test CAUSED BY: TableLoadingException: Failed to load metadata for table: test CAUSED BY: InvalidStorageDescriptor Exception: com.cloudera.impala.common.AnalysisException: DATE/DATETIME/TIMESTAMP literals not supported: 2009-01-01 00:00:00 CAUSED BY: AnalysisException: DATE/DATETIME/TIMESTAMP literals not supported: 2009-01-01 00:00:00 [localhost:21000] > create table t (i int) partitioned by (t timestamp); [localhost:21000] > insert into t partition(t='abcd') select 1 from functional.alltypes; [localhost:21000] > select * from t; Error communicating with impalad: None [Not connected] > Goodbye -- This won't actually insert any data [localhost:21000] > insert into t partition(t='abcd') select 1; [localhost:21000] > select * from t; Error communicating with impalad: None [Not connected] > Goodbye
Impalad log - invalid timestamp value
I0412 08:26:55.006533 17952 status.cc:42] InternalException: Error updating metastore
CAUSED BY: InvalidObjectException: incomplete partition name - missing t
@ 0xd227c2 impala::Status::Status()
@ 0xafaa7e impala::ImpalaServer::UpdateMetastore()
@ 0xaec2bd impala::ImpalaServer::QueryExecState::UpdateMetastore()
@ 0xaeadff impala::ImpalaServer::QueryExecState::Wait()
@ 0xafa727 impala::ImpalaServer::Wait()
@ 0xb7ed82 boost::_mfi::mf1<>::operator()()
@ 0xb7ecab boost::_bi::list2<>::operator()<>()
@ 0xb7ec2d boost::_bi::bind_t<>::operator()()
@ 0xb7ebc0 boost::detail::thread_data<>::run()
@ 0x7fbefd7c4ce9 (unknown)
@ 0x7fbefd5a2e9a start_thread
@ 0x7fbefb951cbd (unknown)
Failed DCHECK when attempting to scan the table:
Tuple(id=0 size=24 slots=[Slot(id=0 type=TIMESTAMP col=0 offset=8 null=(offset=0 mask=2)), Slot(id=1 type=INT col=1 offset=4 null=(offset=0 mask=1))]) F0412 07:26:25.754392 30150 llvm-codegen.cc:282] Check failed: false Invalid type. *** Check failure stack trace: *** @ 0x16d948d google::LogMessage::Fail() @ 0x16dbc1f google::LogMessage::SendToLog() @ 0x16d908b google::LogMessage::Flush() @ 0x16dc4ad google::LogMessageFatal::~LogMessageFatal() @ 0xb8967f impala::LlvmCodeGen::GetType() @ 0xc9c4dd impala::TupleDescriptor::GenerateLlvmStruct() @ 0xe64147 impala::TextConverter::CodegenWriteSlot() @ 0xe78769 impala::HdfsScanner::CodegenWriteCompleteTuple() @ 0xe42e71 impala::HdfsTextScanner::Codegen() @ 0xe10437 impala::HdfsScanNode::Prepare() @ 0xcbafc9 impala::PlanFragmentExecutor::Prepare() @ 0xaec5a0 impala::ImpalaServer::FragmentExecState::Prepare() @ 0xaff984 impala::ImpalaServer::StartPlanFragmentExecution() @ 0xafee3b impala::ImpalaServer::ExecPlanFragment() @ 0xd1e698 impala::ImpalaInternalServiceProcessor::process_ExecPlanFragment() @ 0xd1e3cf impala::ImpalaInternalServiceProcessor::dispatchCall() @ 0xb06914 apache::thrift::TDispatchProcessor::process() @ 0x16b753d apache::thrift::server::TThreadedServer::Task::run() @ 0x16b9736 apache::thrift::concurrency::PthreadThread::threadMain() @ 0x7f960524ce9a start_thread @ 0x7f96035fbcbd (unknown) Aborted (core dumped)
Performing the same set of steps but issuing a refresh before the select * (forcing the metadata to be reloaded) results in:
[localhost:21000] > select * from test;
Query: select * from test
ERROR: com.cloudera.impala.common.AnalysisException: Analysis exception (in select * from test)
at com.cloudera.impala.analysis.AnalysisContext.analyze(AnalysisContext.java:201)
at com.cloudera.impala.service.Frontend.createExecRequest(Frontend.java:375)
at com.cloudera.impala.service.JniFrontend.createExecRequest(JniFrontend.java:125)
Caused by: com.cloudera.impala.catalog.Db$TableLoadingException: Failed to load metadata for table: test
at com.cloudera.impala.catalog.HdfsTable.load(HdfsTable.java:718)
at com.cloudera.impala.catalog.Table.load(Table.java:140)
at com.cloudera.impala.catalog.Db$LazyTableMap.loadTable(Db.java:227)
at com.cloudera.impala.catalog.Db$LazyTableMap.access$000(Db.java:110)
at com.cloudera.impala.catalog.Db$LazyTableMap$1.load(Db.java:119)
at com.cloudera.impala.catalog.Db$LazyTableMap$1.load(Db.java:116)
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3568)
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350)
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2313)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228)
at com.google.common.cache.LocalCache.get(LocalCache.java:3965)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3969)
at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4829)
at com.cloudera.impala.catalog.Db$LazyTableMap.get(Db.java:193)
at com.cloudera.impala.catalog.Db.getTable(Db.java:300)
at com.cloudera.impala.analysis.Analyzer.registerBaseTableRef(Analyzer.java:176)
... 5 more
Caused by: com.cloudera.impala.catalog.HdfsStorageDescriptor$InvalidStorageDescriptorException: com.cloudera.impala.common.AnalysisException: DATE/DATETIME/TIMESTAMP literals not supported: abc
at com.cloudera.impala.catalog.HdfsTable.loadPartitions(HdfsTable.java:601)
at com.cloudera.impala.catalog.HdfsTable.load(HdfsTable.java:710)
... 20 more
Caused by: com.cloudera.impala.common.AnalysisException: DATE/DATETIME/TIMESTAMP literals not supported: abc
at com.cloudera.impala.analysis.LiteralExpr.create(LiteralExpr.java:45)
at com.cloudera.impala.catalog.HdfsTable.loadPartitions(HdfsTable.java:595)
... 21 more