Details
-
Sub-task
-
Status: Resolved
-
Major
-
Resolution: Won't Fix
-
None
-
None
-
None
Description
When building filtered partition path, Tajo need to check the path exists on file system. Currently, if the partition exists on catalog and it doesn't exist on file system, users would fail to execute a count all query as following:
2015-09-17 15:54:50,219 ERROR org.apache.tajo.master.GlobalEngine: Stack Trace: org.apache.tajo.exception.TajoInternalError: internal error: File does not exist: /tajo/warehouse/tpch_partition/customer/c_nationkey=9 at org.apache.hadoop.hdfs.server.namenode.FSDirectory.getContentSummary(FSDirectory.java:2157) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getContentSummary(FSNamesystem.java:4398) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getContentSummary(NameNodeRpcServer.java:998) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getContentSummary(ClientNamenodeProtocolServerSideTranslatorPB.java:855) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033) at org.apache.tajo.plan.rewrite.rules.PartitionedTableRewriter.updateTableStat(PartitionedTableRewriter.java:439) at org.apache.tajo.plan.rewrite.rules.PartitionedTableRewriter.access$100(PartitionedTableRewriter.java:49) at org.apache.tajo.plan.rewrite.rules.PartitionedTableRewriter$Rewriter.visitScan(PartitionedTableRewriter.java:523) at org.apache.tajo.plan.rewrite.rules.PartitionedTableRewriter$Rewriter.visitScan(PartitionedTableRewriter.java:508) at org.apache.tajo.plan.visitor.BasicLogicalPlanVisitor.visit(BasicLogicalPlanVisitor.java:110) at org.apache.tajo.plan.visitor.BasicLogicalPlanVisitor.visitGroupBy(BasicLogicalPlanVisitor.java:221) at org.apache.tajo.plan.visitor.BasicLogicalPlanVisitor.visit(BasicLogicalPlanVisitor.java:83) at org.apache.tajo.plan.visitor.BasicLogicalPlanVisitor.visitRoot(BasicLogicalPlanVisitor.java:163) at org.apache.tajo.plan.visitor.BasicLogicalPlanVisitor.visit(BasicLogicalPlanVisitor.java:62) at org.apache.tajo.plan.rewrite.rules.PartitionedTableRewriter.rewrite(PartitionedTableRewriter.java:82)
Attachments
Issue Links
- is related to
-
TAJO-1493 Make partition pruning based on catalog informations
- Resolved