Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.13.0
    • Fix Version/s: 0.13.0
    • Component/s: None
    • Labels:

      Description

      1. HIVE-6375.4.patch
        16 kB
        Szehon Ho
      2. HIVE-6375.3.patch
        16 kB
        Szehon Ho
      3. HIVE-6375.2.patch
        16 kB
        Szehon Ho
      4. HIVE-6375.patch
        9 kB
        Szehon Ho

        Issue Links

          Activity

          Hide
          Lefty Leverenz added a comment -

          Great, I'll revise the Parquet doc. Thanks Szehon Ho.

          Show
          Lefty Leverenz added a comment - Great, I'll revise the Parquet doc. Thanks Szehon Ho .
          Hide
          Szehon Ho added a comment -

          Lefty Leverenz I believe HIVE-6938 adds support for parquet column-rename (with a flag). I added a link.

          Show
          Szehon Ho added a comment - Lefty Leverenz I believe HIVE-6938 adds support for parquet column-rename (with a flag). I added a link.
          Hide
          Lefty Leverenz added a comment -

          Support for CTAS is documented in the wiki:

          If there's another JIRA ticket for column rename, please link it to this ticket. For now, the wiki continues to cite this ticket for column rename.

          Show
          Lefty Leverenz added a comment - Support for CTAS is documented in the wiki: Language Manual – Parquet – Limitations If there's another JIRA ticket for column rename, please link it to this ticket. For now, the wiki continues to cite this ticket for column rename.
          Hide
          Xuefu Zhang added a comment -

          Previous commit didn't seem going thru. Tried again.

          Show
          Xuefu Zhang added a comment - Previous commit didn't seem going thru. Tried again.
          Hide
          Xuefu Zhang added a comment -

          Patch committed to trunk. Thanks to Szehon for the contribution.

          Show
          Xuefu Zhang added a comment - Patch committed to trunk. Thanks to Szehon for the contribution.
          Hide
          Xuefu Zhang added a comment -

          +1

          Show
          Xuefu Zhang added a comment - +1
          Hide
          Szehon Ho added a comment -

          Xuefu Zhang Can you please take a look? Thanks.

          Show
          Szehon Ho added a comment - Xuefu Zhang Can you please take a look? Thanks.
          Hide
          Hive QA added a comment -

          Overall: +1 all checks pass

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12630586/HIVE-6375.4.patch

          SUCCESS: +1 5177 tests passed

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1483/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1483/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          

          This message is automatically generated.

          ATTACHMENT ID: 12630586

          Show
          Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12630586/HIVE-6375.4.patch SUCCESS: +1 5177 tests passed Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1483/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1483/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated. ATTACHMENT ID: 12630586
          Hide
          Szehon Ho added a comment -

          Looks like 'database:default' output was added to all test output files by recent commit HIVE-5958. Regenerating my new test output due to the same.

          Show
          Szehon Ho added a comment - Looks like 'database:default' output was added to all test output files by recent commit HIVE-5958 . Regenerating my new test output due to the same.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12630220/HIVE-6375.3.patch

          ERROR: -1 due to 2 failed/errored test(s), 5176 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_parquet_ctas
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority
          

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1449/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1449/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 2 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12630220

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12630220/HIVE-6375.3.patch ERROR: -1 due to 2 failed/errored test(s), 5176 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_parquet_ctas org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1449/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1449/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed This message is automatically generated. ATTACHMENT ID: 12630220
          Hide
          Szehon Ho added a comment -

          Test only fixes. Last change did not completely guaratnee the test output, need to sort by both key,value. Switched to another test table that has unique key,value pair.

          Other tests insert_into5 and insert_into6 are known issues (HIVE-6479). Final failure on minimr driver is not reproducible.

          Show
          Szehon Ho added a comment - Test only fixes. Last change did not completely guaratnee the test output, need to sort by both key,value. Switched to another test table that has unique key,value pair. Other tests insert_into5 and insert_into6 are known issues ( HIVE-6479 ). Final failure on minimr driver is not reproducible.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12629901/HIVE-6375.2.patch

          ERROR: -1 due to 4 failed/errored test(s), 5164 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_parquet_ctas
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket_num_reducers
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_insert_into5
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_insert_into6
          

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1430/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1430/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 4 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12629901

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12629901/HIVE-6375.2.patch ERROR: -1 due to 4 failed/errored test(s), 5164 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_parquet_ctas org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket_num_reducers org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_insert_into5 org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_insert_into6 Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1430/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1430/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 4 tests failed This message is automatically generated. ATTACHMENT ID: 12629901
          Hide
          Szehon Ho added a comment -

          Yea, looks like a similar issue.

          Show
          Szehon Ho added a comment - Yea, looks like a similar issue.
          Hide
          Mohammad Kamrul Islam added a comment -

          +1
          reviewed the patch.

          CTAS for aver doesn't work for the same reason (HIVE-5803).
          Hopefully, the patch will help avro as well.

          Show
          Mohammad Kamrul Islam added a comment - +1 reviewed the patch. CTAS for aver doesn't work for the same reason ( HIVE-5803 ). Hopefully, the patch will help avro as well.
          Hide
          Szehon Ho added a comment -

          Thanks Xuefu for review. Incorporated feedback and fixed test output. Seems select from srcbucket has some randomness to the result, as it is a bucketed table.

          Show
          Szehon Ho added a comment - Thanks Xuefu for review. Incorporated feedback and fixed test output. Seems select from srcbucket has some randomness to the result, as it is a bucketed table.
          Hide
          Xuefu Zhang added a comment -

          Patch looks good. Minor comment on RB. The test diff needs to be fixed.

          Show
          Xuefu Zhang added a comment - Patch looks good. Minor comment on RB. The test diff needs to be fixed.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12629690/HIVE-6375.patch

          ERROR: -1 due to 3 failed/errored test(s), 5134 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ctas_hadoop20
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_merge3
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_parquet_ctas
          

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1410/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1410/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 3 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12629690

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12629690/HIVE-6375.patch ERROR: -1 due to 3 failed/errored test(s), 5134 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ctas_hadoop20 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_merge3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_parquet_ctas Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1410/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1410/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 3 tests failed This message is automatically generated. ATTACHMENT ID: 12629690
          Hide
          Szehon Ho added a comment -

          As ParquetSerde is resolving columns by name, column rename will not work for parquet file as of now. Renaming the JIRA to reflect this.

          Show
          Szehon Ho added a comment - As ParquetSerde is resolving columns by name, column rename will not work for parquet file as of now. Renaming the JIRA to reflect this.
          Hide
          Szehon Ho added a comment -

          There is a bug in SemanticAnalyzer that leads to different column names used in FileSink and CreateTable operators. Submitting a fix.

          Show
          Szehon Ho added a comment - There is a bug in SemanticAnalyzer that leads to different column names used in FileSink and CreateTable operators. Submitting a fix.

            People

            • Assignee:
              Szehon Ho
              Reporter:
              Brock Noland
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development