Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-504

when inserting to a column partitioned table, if a queryunit attempt fails, an AlreadyExistsStorageException will throw

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: 0.8.0
    • Component/s: distributed query plan
    • Labels:
      None

      Description

      I came across such exception these days.

      ColumnPartitionedTableStoreExec creates hdfs directories based on partition values. For example is one of the partitions is 'col1=2', a directory named 'col1=2' will be created. If query unit attempt fails, that directory still exist. After that TajoQueryMaster start another query unit attempt to reprocessing the logic in ColumnPartitionedTableStoreExec. This physical executor try to create 'col1=2' again, but before creating, it found the target directory already exist, it will throw an AlreadyExistsStorageException. And eventually, the query fails.

        Issue Links

          Activity

          Hide
          hyunsik Hyunsik Choi added a comment -

          The main cause of this issue are two things.

          One is there is no output committer. In Tajo, when some task is failed, QueryMaster retries the task on another node. But, if the task belongs to the final step that generates output files in HDFS, this error occurs because the final output file already exists. In order to solve this problem, we have to implement some kind of output committer which would be similar to that of MR.

          The second one is the bug of ColumnPartitionedTableStoreExec. During few weeks, I have added sort-based column partitioned table executor. which is a default executor for wring column partiions. It eliminates many problems and it works well. I think that the second one was resolved.

          So, the remain issue is only output committer. I'll create the issue in order to clearfy what we should do, and I close this issue as not a problem.

          Thank you Min for this report!

          Show
          hyunsik Hyunsik Choi added a comment - The main cause of this issue are two things. One is there is no output committer. In Tajo, when some task is failed, QueryMaster retries the task on another node. But, if the task belongs to the final step that generates output files in HDFS, this error occurs because the final output file already exists. In order to solve this problem, we have to implement some kind of output committer which would be similar to that of MR. The second one is the bug of ColumnPartitionedTableStoreExec. During few weeks, I have added sort-based column partitioned table executor. which is a default executor for wring column partiions. It eliminates many problems and it works well. I think that the second one was resolved. So, the remain issue is only output committer. I'll create the issue in order to clearfy what we should do, and I close this issue as not a problem. Thank you Min for this report!

            People

            • Assignee:
              Unassigned
              Reporter:
              coderplay Min Zhou
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development