Hive
  1. Hive
  2. HIVE-6022

Load statements with incorrect order of partitions put input files to unreadable places

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.13.0
    • Component/s: Query Processor
    • Labels:
      None

      Description

      Load statements with incorrect order of partitions put input files to incorrect paths.

      CREATE TABLE test_parts (c1 string, c2 int) PARTITIONED BY (p1 string,p2 string);
      LOAD DATA LOCAL INPATH '/opt/hive/examples/files/kv1.txt' OVERWRITE INTO TABLE test_parts PARTITION (p2='p1', p1='p2')"
      

      The input file is located as below and the data is not readable.

      % find /user/hive/warehouse/test_parts/
      /user/hive/warehouse/test_parts/
      /user/hive/warehouse/test_parts//p1=p2
      /user/hive/warehouse/test_parts//p1=p2/p2=p1
      /user/hive/warehouse/test_parts//p2=p1
      /user/hive/warehouse/test_parts//p2=p1/p1=p2
      /user/hive/warehouse/test_parts//p2=p1/p1=p2/.kv1.txt.crc
      /user/hive/warehouse/test_parts//p2=p1/p1=p2/kv1.txt
      
      1. HIVE-6022.1.patch
        5 kB
        Xuefu Zhang
      2. HIVE-6022.1.patch.txt
        5 kB
        Teruyoshi Zenmyo
      3. HIVE-6022.2.patch.txt
        6 kB
        Teruyoshi Zenmyo

        Activity

        Hide
        Xuefu Zhang added a comment -

        Patch committed to trunk. Thanks to Teruyoshi for the contribution.

        Show
        Xuefu Zhang added a comment - Patch committed to trunk. Thanks to Teruyoshi for the contribution.
        Hide
        Xuefu Zhang added a comment -

        Teruyoshi Zenmyo Thanks for fixing the test cases. The patch looks good now. Will commit in a day.

        Show
        Xuefu Zhang added a comment - Teruyoshi Zenmyo Thanks for fixing the test cases. The patch looks good now. Will commit in a day.
        Hide
        Hive QA added a comment -

        Overall: +1 all checks pass

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12618588/HIVE-6022.2.patch.txt

        SUCCESS: +1 4786 tests passed

        Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/631/testReport
        Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/631/console

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        

        This message is automatically generated.

        ATTACHMENT ID: 12618588

        Show
        Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12618588/HIVE-6022.2.patch.txt SUCCESS: +1 4786 tests passed Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/631/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/631/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated. ATTACHMENT ID: 12618588
        Hide
        Teruyoshi Zenmyo added a comment -

        Fixes for the failed tests

        • validate partSpec before sorting according to partition keys.
        • modify dyn_part4.q.out
        Show
        Teruyoshi Zenmyo added a comment - Fixes for the failed tests validate partSpec before sorting according to partition keys. modify dyn_part4.q.out
        Hide
        Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12618542/HIVE-6022.1.patch

        ERROR: -1 due to 3 failed/errored test(s), 4786 tests executed
        Failed tests:

        org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_analyze1
        org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_dyn_part1
        org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_dyn_part4
        

        Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/629/testReport
        Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/629/console

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 3 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12618542

        Show
        Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12618542/HIVE-6022.1.patch ERROR: -1 due to 3 failed/errored test(s), 4786 tests executed Failed tests: org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_analyze1 org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_dyn_part1 org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_dyn_part4 Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/629/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/629/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 3 tests failed This message is automatically generated. ATTACHMENT ID: 12618542
        Hide
        Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12618411/HIVE-6022.1.patch.txt

        ERROR: -1 due to 3 failed/errored test(s), 4786 tests executed
        Failed tests:

        org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_analyze1
        org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_dyn_part1
        org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_dyn_part4
        

        Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/628/testReport
        Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/628/console

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 3 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12618411

        Show
        Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12618411/HIVE-6022.1.patch.txt ERROR: -1 due to 3 failed/errored test(s), 4786 tests executed Failed tests: org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_analyze1 org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_dyn_part1 org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_dyn_part4 Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/628/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/628/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 3 tests failed This message is automatically generated. ATTACHMENT ID: 12618411
        Hide
        Xuefu Zhang added a comment -

        Rename the patch so that the test can run.

        Show
        Xuefu Zhang added a comment - Rename the patch so that the test can run.
        Hide
        Xuefu Zhang added a comment -

        +1, patch looks good to me.

        Show
        Xuefu Zhang added a comment - +1, patch looks good to me.
        Hide
        Teruyoshi Zenmyo added a comment -

        Thanks for taking care of this. I have created a review board entry.
        https://reviews.apache.org/r/16239/

        Show
        Teruyoshi Zenmyo added a comment - Thanks for taking care of this. I have created a review board entry. https://reviews.apache.org/r/16239/
        Hide
        Xuefu Zhang added a comment -

        Teruyoshi Zenmyo Thanks for your contribution. Could you please put a review board entry here?

        Show
        Xuefu Zhang added a comment - Teruyoshi Zenmyo Thanks for your contribution. Could you please put a review board entry here?

          People

          • Assignee:
            Teruyoshi Zenmyo
            Reporter:
            Teruyoshi Zenmyo
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development