[TAJO-2047] Implement S3TableSpace::commitTable.

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Component/s: S3, Storage
• Assignee: JaeHwa Jung
• Reporter: JaeHwa Jung
• Created: 08/Jan/16 07:12
• Updated: 11/Jan/16 09:19
• Resolved: 11/Jan/16 09:18

Description

Query hangs while executing CTAS for partition table on aws as following:

• Root directory exists on s3
• Dataset : tpc-h 1G
• Query

  CREATE TABLE LINEITEM_P (L_ORDERKEY bigint, L_PARTKEY bigint, L_SUPPKEY bigint, L_LINENUMBER bigint, L_QUANTITY double, L_EXTENDEDPRICE double, L_DISCOUNT double, L_TAX double, L_RETURNFLAG text, L_LINESTATUS text, L_COMMITDATE text, L_RECEIPTDATE text, L_SHIPINSTRUCT text, L_SHIPMODE text, L_COMMENT text) 
  USING TEXT WITH ('text.delimiter'='|')
  PARTITION BY COLUMN (L_SHIPDATE text) AS 
  SELECT L_ORDERKEY, L_PARTKEY, L_SUPPKEY, L_LINENUMBER, L_QUANTITY, L_EXTENDEDPRICE, L_DISCOUNT, L_TAX, L_RETURNFLAG, L_LINESTATUS, L_COMMITDATE, L_RECEIPTDATE, L_SHIPINSTRUCT, L_SHIPMODE, L_COMMENT, L_SHIPDATE FROM LINEITEM;
  

• TSql logs

  Progress: 99%, response time: 1157.913 sec
  Progress: 99%, response time: 1158.914 sec
  Progress: 100%, response time: 1159.914 sec
  Progress: 100%, response time: 1160.915 sec
  Progress: 100%, response time: 1161.916 sec
  Progress: 100%, response time: 1162.917 sec
  Progress: 100%, response time: 1163.917 sec
  Progress: 100%, response time: 1164.918 sec
  Progress: 100%, response time: 1165.919 sec
  Progress: 100%, response time: 1166.919 sec
  Progress: 100%, response time: 1167.92 sec
  Progress: 100%, response time: 1168.921 sec
  Progress: 100%, response time: 1169.921 sec
  Progress: 100%, response time: 1170.922 sec
  Progress: 100%, response time: 1171.923 sec
  Progress: 100%, response time: 1172.924 sec
  Progress: 100%, response time: 1173.924 sec
  Progress: 100%, response time: 1174.925 sec
  

• TajoWorker logs

  2016-01-08 07:50:56,158 INFO org.apache.tajo.querymaster.Stage: Stage completed - eb_1452238256580_0001_000002 (total=6, success=6, killed=0)
  2016-01-08 07:50:56,159 INFO org.apache.tajo.querymaster.Query: Processing q_1452238256580_0001 of type STAGE_COMPLETED
  2016-01-08 07:50:56,159 INFO org.apache.tajo.engine.planner.global.ParallelExecutionQueue: Next executable block eb_1452238256580_0001_000003
  2016-01-08 07:50:56,159 INFO org.apache.tajo.querymaster.Query: Complete Stage[eb_1452238256580_0001_000002], State: SUCCEEDED, 2/2. 
  2016-01-08 07:50:56,159 INFO org.apache.tajo.querymaster.Query: Processing q_1452238256580_0001 of type QUERY_COMPLETED
  2016-01-08 07:50:56,161 INFO org.apache.tajo.worker.ExecutionBlockContext: Worker's task counter - total:6, succeeded: 6, killed: 0, failed: 0
  2016-01-08 07:50:56,194 INFO org.apache.tajo.worker.TaskManager: Stopped execution block:eb_1452238256580_0001_000002
  

I found that query hangs on FileTableSpace::commitTable. We need to implement commitTable in S3TableSpace by considering s3 features.

Issue Links

• is related to: TAJO-1905 Insert clause to partitioned table fails on S3