[TAJO-1905] Insert clause to partitioned table fails on S3

• Type: Sub-task
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: 0.13.0
• Component/s: QueryMaster, S3
• Assignee: JaeHwa Jung
• Reporter: Jinho Kim
• Created: 02/Oct/15 10:07
• Updated: 04/Dec/17 00:00

Description

Here is the error log

2015-10-02 18:54:40,399 ERROR org.apache.hadoop.fs.s3a.S3AFileSystem: rename: src not found s3a://bucket/tpch-1g-p/lineitem/.staging/q_1443779192380_0001/RESULT/l_shipdate=1996-01-30
2015-10-02 18:54:51,357 ERROR org.apache.hadoop.fs.s3a.S3AFileSystem: rename: src not found s3a://bucket/tpch-1g-p/lineitem/.staging/q_1443779192380_0001/RESULT/l_shipdate=1993-11-09
2015-10-02 18:55:03,955 ERROR org.apache.tajo.querymaster.Query: No such file or directory: s3a://bucket/lineitem/l_shipdate=1994-02-02
java.io.FileNotFoundException: No such file or directory: s3a://bucket/tpch-1g-p/lineitem/l_shipdate=1994-02-02
at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:996)
at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:77)
at org.apache.hadoop.fs.FileSystem.getContentSummary(FileSystem.java:1467)
at org.apache.tajo.querymaster.Query$QueryCompletedTransition.getPartitionsWithContentsSummary(Query.java:550)
at org.apache.tajo.querymaster.Query$QueryCompletedTransition.finalizeQuery(Query.java:512)
at org.apache.tajo.querymaster.Query$QueryCompletedTransition.transition(Query.java:446)
at org.apache.tajo.querymaster.Query$QueryCompletedTransition.transition(Query.java:435)
at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
at org.apache.tajo.querymaster.Query.handle(Query.java:874)
at org.apache.tajo.querymaster.Query.handle(Query.java:63)
at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:175)
at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:108)
at java.lang.Thread.run(Thread.java:745)
2015-10-02 18:55:03,958 INFO org.apache.tajo.querymaster.Query: q_1443779192380_0001 Query Transitioned from QUERY_RUNNING to QUERY_ERROR
2015-10-02 18:55:03,958 INFO org.apache.tajo.querymaster.Query: Processing q_1443779192380_0001 of type DIAGNOSTIC_UPDATE
2015-10-02 18:55:03,958 INFO org.apache.tajo.querymaster.QueryMasterTask: Query completion notified from q_1443779192380_0001 final state: QUERY_ERROR
2015-10-02 18:55:03,960 INFO org.apache.tajo.querymaster.QueryMasterTask: Stopping QueryMasterTask:q_1443779192380_0001
2015-10-02 18:55:03,960 INFO org.apache.tajo.querymaster.QueryMasterTask: Cleanup resources of all workers. Query: q_1443779192380_0001, workers: 1
2015-10-02 18:55:03,962 INFO org.apache.tajo.querymaster.QueryMasterTask: Stopped QueryMasterTask:q_1443779192380_0001

Issue Links

• depends upon: TAJO-2069 Implement finding the total size of all objects in a bucket with AWS SDK.
• depends upon: TAJO-2063 Refactor FileTablespace::commitOutputData.
• relates to: TAJO-2047 Implement S3TableSpace::commitTable.