[TAJO-1865] When building filtered partition path, check partition directory exists on filesystem.

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Component/s: Planner/Optimizer
• Assignee: JaeHwa Jung
• Reporter: JaeHwa Jung
• Created: 17/Sep/15 07:21
• Updated: 17/Sep/15 11:07
• Resolved: 17/Sep/15 11:07

Description

When building filtered partition path, Tajo need to check the path exists on file system. Currently, if the partition exists on catalog and it doesn't exist on file system, users would fail to execute a count all query as following:

2015-09-17 15:54:50,219 ERROR org.apache.tajo.master.GlobalEngine: 
Stack Trace:
org.apache.tajo.exception.TajoInternalError: internal error: File does not exist: /tajo/warehouse/tpch_partition/customer/c_nationkey=9
	at org.apache.hadoop.hdfs.server.namenode.FSDirectory.getContentSummary(FSDirectory.java:2157)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getContentSummary(FSNamesystem.java:4398)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getContentSummary(NameNodeRpcServer.java:998)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getContentSummary(ClientNamenodeProtocolServerSideTranslatorPB.java:855)
	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)

	at org.apache.tajo.plan.rewrite.rules.PartitionedTableRewriter.updateTableStat(PartitionedTableRewriter.java:439)
	at org.apache.tajo.plan.rewrite.rules.PartitionedTableRewriter.access$100(PartitionedTableRewriter.java:49)
	at org.apache.tajo.plan.rewrite.rules.PartitionedTableRewriter$Rewriter.visitScan(PartitionedTableRewriter.java:523)
	at org.apache.tajo.plan.rewrite.rules.PartitionedTableRewriter$Rewriter.visitScan(PartitionedTableRewriter.java:508)
	at org.apache.tajo.plan.visitor.BasicLogicalPlanVisitor.visit(BasicLogicalPlanVisitor.java:110)
	at org.apache.tajo.plan.visitor.BasicLogicalPlanVisitor.visitGroupBy(BasicLogicalPlanVisitor.java:221)
	at org.apache.tajo.plan.visitor.BasicLogicalPlanVisitor.visit(BasicLogicalPlanVisitor.java:83)
	at org.apache.tajo.plan.visitor.BasicLogicalPlanVisitor.visitRoot(BasicLogicalPlanVisitor.java:163)
	at org.apache.tajo.plan.visitor.BasicLogicalPlanVisitor.visit(BasicLogicalPlanVisitor.java:62)
	at org.apache.tajo.plan.rewrite.rules.PartitionedTableRewriter.rewrite(PartitionedTableRewriter.java:82)

Issue Links

• is related to: TAJO-1493 Make partition pruning based on catalog informations