Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-1384

Duplicated output file path problem

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.11.0, 0.10.1
    • Component/s: Storage
    • Labels:
      None

      Description

      Introduction
      When consecutively inserting data into an existing directory, each files must have different names. To avoid the duplicated name problem, a unique sequence number is appended to the end of the file name. In addition, the max sequence number of pre-existing files is first found before starting writing to the final output directory.

      Problem
      When finding the max sequence number of pre-existing files, those files are not sorted by their sequence numbers. This finally causes the duplicated file name problem.

        Activity

        Hide
        githubbot ASF GitHub Bot added a comment -

        GitHub user jihoonson opened a pull request:

        https://github.com/apache/tajo/pull/406

        TAJO-1384: Duplicated output file path problem

        You can merge this pull request into a Git repository by running:

        $ git pull https://github.com/jihoonson/tajo-2 TAJO-1384

        Alternatively you can review and apply these changes as the patch at:

        https://github.com/apache/tajo/pull/406.patch

        To close this pull request, make a commit to your master/trunk branch
        with (at least) the following in the commit message:

        This closes #406


        commit afc25a285e6ce7ca999971d9e53e365b698a2d91
        Author: Jihoon Son <jihoonson@apache.org>
        Date: 2015-03-10T06:36:04Z

        TAJO-1384


        Show
        githubbot ASF GitHub Bot added a comment - GitHub user jihoonson opened a pull request: https://github.com/apache/tajo/pull/406 TAJO-1384 : Duplicated output file path problem You can merge this pull request into a Git repository by running: $ git pull https://github.com/jihoonson/tajo-2 TAJO-1384 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/tajo/pull/406.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #406 commit afc25a285e6ce7ca999971d9e53e365b698a2d91 Author: Jihoon Son <jihoonson@apache.org> Date: 2015-03-10T06:36:04Z TAJO-1384
        Hide
        hyunsik Hyunsik Choi added a comment -

        just marked as PATCH AVAILABLE.

        Show
        hyunsik Hyunsik Choi added a comment - just marked as PATCH AVAILABLE.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user hyunsik commented on the pull request:

        https://github.com/apache/tajo/pull/406#issuecomment-78130692

        The change looks straightforward. The second patch that eliminates unnecessary path list and unnecessary sort looks much better than the first one.

        If all unit tests are passed, I'll finish the review.

        Show
        githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on the pull request: https://github.com/apache/tajo/pull/406#issuecomment-78130692 The change looks straightforward. The second patch that eliminates unnecessary path list and unnecessary sort looks much better than the first one. If all unit tests are passed, I'll finish the review.
        Hide
        tajoqa Tajo QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12703611/TAJO-1384.patch
        against master revision release-0.9.0-rc0-194-g13316bd.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

        +1 checkstyle. The patch generated 0 code style errors.

        -1 findbugs. The patch appears to introduce 10 new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in tajo-storage/tajo-storage-common tajo-storage/tajo-storage-hdfs.

        Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/608//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/608//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-storage-hdfs.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/608//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-storage-common.html
        Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/608//console

        This message is automatically generated.

        Show
        tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12703611/TAJO-1384.patch against master revision release-0.9.0-rc0-194-g13316bd. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to introduce 10 new Findbugs (version 2.0.3) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in tajo-storage/tajo-storage-common tajo-storage/tajo-storage-hdfs. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/608//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/608//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-storage-hdfs.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/608//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-storage-common.html Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/608//console This message is automatically generated.
        Hide
        hyunsik Hyunsik Choi added a comment -

        +1 ship it.

        Show
        hyunsik Hyunsik Choi added a comment - +1 ship it.
        Hide
        jihoonson Jihoon Son added a comment -

        Thanks for quick review!

        Show
        jihoonson Jihoon Son added a comment - Thanks for quick review!
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Tajo-master-build #608 (See https://builds.apache.org/job/Tajo-master-build/608/)
        TAJO-1384: Duplicated output file path problem. (jihoon) (jihoonson: rev 513a8c3086b889d9b701d5c97eda3fb8c5453834)

        • tajo-storage/tajo-storage-common/src/main/java/org/apache/tajo/storage/StorageUtil.java
        • tajo-storage/tajo-storage-hdfs/src/test/java/org/apache/tajo/storage/TestStorageUtil.java
        • CHANGES
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Tajo-master-build #608 (See https://builds.apache.org/job/Tajo-master-build/608/ ) TAJO-1384 : Duplicated output file path problem. (jihoon) (jihoonson: rev 513a8c3086b889d9b701d5c97eda3fb8c5453834) tajo-storage/tajo-storage-common/src/main/java/org/apache/tajo/storage/StorageUtil.java tajo-storage/tajo-storage-hdfs/src/test/java/org/apache/tajo/storage/TestStorageUtil.java CHANGES
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Tajo-master-CODEGEN-build #246 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/246/)
        TAJO-1384: Duplicated output file path problem. (jihoon) (jihoonson: rev 513a8c3086b889d9b701d5c97eda3fb8c5453834)

        • tajo-storage/tajo-storage-hdfs/src/test/java/org/apache/tajo/storage/TestStorageUtil.java
        • tajo-storage/tajo-storage-common/src/main/java/org/apache/tajo/storage/StorageUtil.java
        • CHANGES
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Tajo-master-CODEGEN-build #246 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/246/ ) TAJO-1384 : Duplicated output file path problem. (jihoon) (jihoonson: rev 513a8c3086b889d9b701d5c97eda3fb8c5453834) tajo-storage/tajo-storage-hdfs/src/test/java/org/apache/tajo/storage/TestStorageUtil.java tajo-storage/tajo-storage-common/src/main/java/org/apache/tajo/storage/StorageUtil.java CHANGES
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user jihoonson closed the pull request at:

        https://github.com/apache/tajo/pull/406

        Show
        githubbot ASF GitHub Bot added a comment - Github user jihoonson closed the pull request at: https://github.com/apache/tajo/pull/406

          People

          • Assignee:
            jihoonson Jihoon Son
            Reporter:
            jihoonson Jihoon Son
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development