Hive
  1. Hive
  2. HIVE-5976

Decouple input formats from STORED as keywords

    Details

    • Type: Task Task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.14.0
    • Component/s: None
    • Labels:
      None

      Description

      As noted in HIVE-5783, we hard code the input formats mapped to keywords. It'd be nice if there was a registration system so we didn't need to do that.

      1. HIVE-5976.9.patch
        95 kB
        David Chen
      2. HIVE-5976.8.patch
        93 kB
        David Chen
      3. HIVE-5976.7.patch
        93 kB
        David Chen
      4. HIVE-5976.6.patch
        76 kB
        David Chen
      5. HIVE-5976.5.patch
        76 kB
        David Chen
      6. HIVE-5976.4.patch
        75 kB
        David Chen
      7. HIVE-5976.3.patch
        75 kB
        David Chen
      8. HIVE-5976.3.patch
        73 kB
        David Chen
      9. HIVE-5976.2.patch
        73 kB
        David Chen
      10. HIVE-5976.patch
        76 kB
        Brock Noland
      11. HIVE-5976.patch
        76 kB
        Brock Noland
      12. HIVE-5976.patch
        75 kB
        Brock Noland
      13. HIVE-5976.patch
        75 kB
        Brock Noland

        Issue Links

          Activity

          Hide
          Brock Noland added a comment - - edited

          I have a patch for this using the ServiceLoader facility:

          http://docs.oracle.com/javase/6/docs/api/java/util/ServiceLoader.html

          I will post it tomorrow.

          Show
          Brock Noland added a comment - - edited I have a patch for this using the ServiceLoader facility: http://docs.oracle.com/javase/6/docs/api/java/util/ServiceLoader.html I will post it tomorrow.
          Show
          Brock Noland added a comment - https://reviews.apache.org/r/18224/
          Hide
          Brock Noland added a comment -

          Note that the "generic stored as" was already in the grammar. This just implements it.

          Show
          Brock Noland added a comment - Note that the "generic stored as" was already in the grammar. This just implements it.
          Hide
          Brock Noland added a comment -

          Latest patch from RB.

          Show
          Brock Noland added a comment - Latest patch from RB.
          Hide
          Brock Noland added a comment -

          Trims trailing ws.

          Show
          Brock Noland added a comment - Trims trailing ws.
          Hide
          Szehon Ho added a comment -

          This is nice, makes it easy to add a new native serde, I looked through the RB and am +1

          But patch needs rebase and resubmission for test at this point.

          Show
          Szehon Ho added a comment - This is nice, makes it easy to add a new native serde, I looked through the RB and am +1 But patch needs rebase and resubmission for test at this point.
          Hide
          David Chen added a comment -

          Hi Brock,

          Thank you for working on this! This will make a lot of things easier, including some of the work I am doing to help improve HCatalog's test coverage.

          If you would like, I'd be glad to help out with finishing up this patch.

          Thanks,
          David

          Show
          David Chen added a comment - Hi Brock, Thank you for working on this! This will make a lot of things easier, including some of the work I am doing to help improve HCatalog's test coverage. If you would like, I'd be glad to help out with finishing up this patch. Thanks, David
          Hide
          David Chen added a comment -

          I took a shot at rebasing Brock's patch on the current trunk. Uploading patch for pre-commit testing.

          Show
          David Chen added a comment - I took a shot at rebasing Brock's patch on the current trunk. Uploading patch for pre-commit testing.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12652566/HIVE-5976.2.patch

          ERROR: -1 due to 16 failed/errored test(s), 5671 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_file_format
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ctas
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat3
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_storage_format_descriptor
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_ctas
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority2
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_fileformat_bad_class
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_genericFileFormat
          org.apache.hive.hcatalog.pig.TestOrcHCatLoader.testReadDataPrimitiveTypes
          org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchAbort
          org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchAbortAndCommit
          org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchCommit_Delimited
          org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/601/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/601/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-601/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 16 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12652566

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12652566/HIVE-5976.2.patch ERROR: -1 due to 16 failed/errored test(s), 5671 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_file_format org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ctas org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_storage_format_descriptor org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_ctas org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority2 org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_fileformat_bad_class org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_genericFileFormat org.apache.hive.hcatalog.pig.TestOrcHCatLoader.testReadDataPrimitiveTypes org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchAbort org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchAbortAndCommit org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchCommit_Delimited org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/601/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/601/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-601/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 16 tests failed This message is automatically generated. ATTACHMENT ID: 12652566
          Hide
          Brock Noland added a comment -

          Thank you David Chen! Some of those failures are unrelated... I think the partition wise ones are related though.

          Show
          Brock Noland added a comment - Thank you David Chen ! Some of those failures are unrelated... I think the partition wise ones are related though.
          Hide
          David Chen added a comment -

          Brock Noland No problem! Would you be fine if I picked this up and fix the test failures?

          Show
          David Chen added a comment - Brock Noland No problem! Would you be fine if I picked this up and fix the test failures?
          Hide
          Brock Noland added a comment -

          I'd be delighted!

          Show
          Brock Noland added a comment - I'd be delighted!
          Hide
          Carl Steinbach added a comment -

          David Chen, I'd like to review this and get it committed. Can you post an RB? Thanks!

          Show
          Carl Steinbach added a comment - David Chen , I'd like to review this and get it committed. Can you post an RB? Thanks!
          Hide
          Brock Noland added a comment -

          FYI it looks like in ParseUtils.ensureClassExists I think we should use JavaUtils.getClassLoader().

          Show
          Brock Noland added a comment - FYI it looks like in ParseUtils.ensureClassExists I think we should use JavaUtils.getClassLoader().
          Hide
          David Chen added a comment -

          Thanks for the info, Brock Noland. I am submitting a new patch to run the pre-commit tests again.

          Carl Steinbach, here is the RB: https://reviews.apache.org/r/23153

          Show
          David Chen added a comment - Thanks for the info, Brock Noland . I am submitting a new patch to run the pre-commit tests again. Carl Steinbach , here is the RB: https://reviews.apache.org/r/23153
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12653081/HIVE-5976.3.patch

          ERROR: -1 due to 10 failed/errored test(s), 5672 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_file_format
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ctas
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat3
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_storage_format_descriptor
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_ctas
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_fileformat_bad_class
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_genericFileFormat
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/629/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/629/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-629/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 10 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12653081

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12653081/HIVE-5976.3.patch ERROR: -1 due to 10 failed/errored test(s), 5672 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_file_format org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ctas org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_storage_format_descriptor org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_ctas org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_fileformat_bad_class org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_genericFileFormat Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/629/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/629/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-629/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 10 tests failed This message is automatically generated. ATTACHMENT ID: 12653081
          Hide
          Brock Noland added a comment -

          Some of those failures appear to be related.

          Show
          Brock Noland added a comment - Some of those failures appear to be related.
          Hide
          David Chen added a comment -

          Agreed. The TestCliDriver failures seem to be related and may be caused by the same underlying problem. I am looking into those failures.

          Show
          David Chen added a comment - Agreed. The TestCliDriver failures seem to be related and may be caused by the same underlying problem. I am looking into those failures.
          Hide
          David Chen added a comment -

          I have posted a new patch that should fix some of the tests. Some of the test failures appear to be caused by slightly different output that Hive now prints due to this patch. The changes should fix the following tests:

          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_ctas
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ctas

          Test failed because output files expected old parse tree dump.

          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_fileformat_bad_class

          Hive now outputs FAILED: SemanticException Cannot find class 'ClassDoesNotExist' if SerDe class is not found.

          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_genericFileFormat

          Hive now prints format name in all caps in quotes.

          I am still looking into the other test failures.

          Show
          David Chen added a comment - I have posted a new patch that should fix some of the tests. Some of the test failures appear to be caused by slightly different output that Hive now prints due to this patch. The changes should fix the following tests: org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_ctas org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ctas Test failed because output files expected old parse tree dump. org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_fileformat_bad_class Hive now outputs FAILED: SemanticException Cannot find class 'ClassDoesNotExist' if SerDe class is not found. org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_genericFileFormat Hive now prints format name in all caps in quotes. I am still looking into the other test failures.
          Hide
          David Chen added a comment -

          Rebasing on trunk.

          Show
          David Chen added a comment - Rebasing on trunk.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12653346/HIVE-5976.4.patch

          ERROR: -1 due to 8 failed/errored test(s), 5673 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_file_format
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat3
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_storage_format_descriptor
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_optimization
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
          org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/650/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/650/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-650/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 8 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12653346

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12653346/HIVE-5976.4.patch ERROR: -1 due to 8 failed/errored test(s), 5673 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_file_format org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_storage_format_descriptor org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_optimization org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/650/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/650/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-650/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 8 tests failed This message is automatically generated. ATTACHMENT ID: 12653346
          Hide
          David Chen added a comment -

          I have fixed the test failure for alter_file_format. The failure was caused by the fact that the SerDe of the table was not set when the StorageFormatDescriptor does not specify a SerDe, as is the case for text and sequencefile. To fix this, I added a new HiveConf variable, HIVEDEFAULTSERDE, which is set to LazySimpleSerDe by default.

          Another source of the failure is if INPUTFORMAT, OUTPUTFORMAT, and SERDE are set using SET FILEFORMAT, the SerDe is not applied.

          Brock Noland - I noticed that AvroSerDe is not included in this patch. Was this because we only want to port the current "native" storage formats for this patch and make Avro a native storage format in a separate patch? I am thinking that AvroSerDe's requirement for a schema URL or schema literal to be set might cause some complications.

          Show
          David Chen added a comment - I have fixed the test failure for alter_file_format. The failure was caused by the fact that the SerDe of the table was not set when the StorageFormatDescriptor does not specify a SerDe, as is the case for text and sequencefile. To fix this, I added a new HiveConf variable, HIVEDEFAULTSERDE, which is set to LazySimpleSerDe by default. Another source of the failure is if INPUTFORMAT, OUTPUTFORMAT, and SERDE are set using SET FILEFORMAT, the SerDe is not applied. Brock Noland - I noticed that AvroSerDe is not included in this patch. Was this because we only want to port the current "native" storage formats for this patch and make Avro a native storage format in a separate patch? I am thinking that AvroSerDe's requirement for a schema URL or schema literal to be set might cause some complications.
          Hide
          Hive QA added a comment -

          Overall: -1 no tests executed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12654861/HIVE-5976.5.patch

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/724/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/724/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-724/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Tests exited with: NonZeroExitCodeException
          Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
          + export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
          + JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
          + export PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
          + PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
          + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
          + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
          + export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
          + M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
          + cd /data/hive-ptest/working/
          + tee /data/hive-ptest/logs/PreCommit-HIVE-Build-724/source-prep.txt
          + [[ false == \t\r\u\e ]]
          + mkdir -p maven ivy
          + [[ svn = \s\v\n ]]
          + [[ -n '' ]]
          + [[ -d apache-svn-trunk-source ]]
          + [[ ! -d apache-svn-trunk-source/.svn ]]
          + [[ ! -d apache-svn-trunk-source ]]
          + cd apache-svn-trunk-source
          + svn revert -R .
          Reverted 'metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java'
          Reverted 'metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java'
          Reverted 'metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java'
          Reverted 'metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java'
          Reverted 'metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java'
          Reverted 'metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java'
          Reverted 'metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java'
          Reverted 'metastore/src/gen/thrift/gen-py/hive_metastore/ttypes.py'
          Reverted 'metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py'
          Reverted 'metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote'
          Reverted 'metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp'
          Reverted 'metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp'
          Reverted 'metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.h'
          Reverted 'metastore/src/gen/thrift/gen-cpp/hive_metastore_types.h'
          Reverted 'metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore_server.skeleton.cpp'
          Reverted 'metastore/src/gen/thrift/gen-rb/thrift_hive_metastore.rb'
          Reverted 'metastore/src/gen/thrift/gen-rb/hive_metastore_types.rb'
          Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PrivilegeBag.java'
          Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ColumnStatistics.java'
          Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Partition.java'
          Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/HiveObjectRef.java'
          Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionsStatsRequest.java'
          Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPrincipalsInRoleResponse.java'
          Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SkewedInfo.java'
          Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ShowCompactResponse.java'
          Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/RequestPartsSpec.java'
          Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AddPartitionsRequest.java'
          Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Type.java'
          Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/HeartbeatTxnRangeResponse.java'
          Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ShowLocksResponse.java'
          Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetOpenTxnsResponse.java'
          Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/OpenTxnsResponse.java'
          Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Schema.java'
          Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java'
          Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PrincipalPrivilegeSet.java'
          Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AddPartitionsResult.java'
          Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/TableStatsResult.java'
          Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/TableStatsRequest.java'
          Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionsByExprResult.java'
          Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Table.java'
          Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/LockRequest.java'
          Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/StorageDescriptor.java'
          Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetRoleGrantsForPrincipalResponse.java'
          Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetOpenTxnsInfoResponse.java'
          Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionsStatsResult.java'
          Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/DropPartitionsResult.java'
          Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Function.java'
          Reverted 'metastore/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php'
          Reverted 'metastore/src/gen/thrift/gen-php/metastore/Types.php'
          Reverted 'metastore/if/hive_metastore.thrift'
          Reverted 'itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestAuthorizationApiAuthorizer.java'
          Reverted 'ql/src/test/results/clientpositive/authorization_role_grant1.q.out'
          Reverted 'ql/src/test/queries/clientpositive/authorization_role_grant1.q'
          Reverted 'ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g'
          Reverted 'ql/src/java/org/apache/hadoop/hive/ql/parse/authorization/HiveAuthorizationTaskFactoryImpl.java'
          Reverted 'ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java'
          Reverted 'ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java'
          Reverted 'ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/sqlstd/SQLStdHiveAccessController.java'
          ++ egrep -v '^X|^Performing status on external'
          ++ awk '{print $2}'
          ++ svn status --no-ignore
          + rm -rf target datanucleus.log ant/target shims/target shims/0.20/target shims/0.20S/target shims/0.23/target shims/aggregator/target shims/common/target shims/common-secure/target packaging/target hbase-handler/target testutils/target jdbc/target metastore/target metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GrantRevokeRoleRequest.java metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GrantRevokeRoleResponse.java metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GrantRevokeType.java itests/target itests/hcatalog-unit/target itests/test-serde/target itests/qtest/target itests/hive-unit-hadoop2/target itests/hive-minikdc/target itests/hive-unit/target itests/custom-serde/target itests/util/target hcatalog/target hcatalog/core/target hcatalog/streaming/target hcatalog/server-extensions/target hcatalog/hcatalog-pig-adapter/target hcatalog/webhcat/svr/target hcatalog/webhcat/java-client/target hwi/target common/target common/src/gen contrib/target service/target serde/target beeline/target odbc/target cli/target ql/dependency-reduced-pom.xml ql/target ql/src/test/results/clientnegative/authorization_role_grant2.q.out ql/src/test/queries/clientnegative/authorization_role_grant2.q
          + svn update
          U    itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniMr.java
          U    itests/qtest/testconfiguration.properties
          U    metastore/if/hive_metastore.thrift
          U    metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp
          U    metastore/src/gen/thrift/gen-cpp/hive_metastore_types.h
          U    metastore/src/gen/thrift/gen-rb/hive_metastore_types.rb
          U    metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Table.java
          U    metastore/src/gen/thrift/gen-php/metastore/Types.php
          U    metastore/src/gen/thrift/gen-py/hive_metastore/ttypes.py
          U    metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java
          U    metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
          A    ql/src/test/queries/clientnegative/temp_table_rename.q
          A    ql/src/test/queries/clientnegative/temp_table_index.q
          A    ql/src/test/queries/clientnegative/temp_table_column_stats.q
          A    ql/src/test/queries/clientnegative/temp_table_create_like_partitions.q
          A    ql/src/test/queries/clientnegative/temp_table_authorize_create_tbl.q
          A    ql/src/test/queries/clientnegative/temp_table_partitions.q
          A    ql/src/test/queries/clientpositive/show_create_table_temp_table.q
          U    ql/src/test/queries/clientpositive/stats19.q
          A    ql/src/test/queries/clientpositive/temp_table_gb1.q
          A    ql/src/test/queries/clientpositive/temp_table_subquery1.q
          A    ql/src/test/queries/clientpositive/temp_table_external.q
          A    ql/src/test/queries/clientpositive/temp_table_names.q
          A    ql/src/test/queries/clientpositive/temp_table_options1.q
          A    ql/src/test/queries/clientpositive/temp_table.q
          A    ql/src/test/queries/clientpositive/temp_table_windowing_expressions.q
          A    ql/src/test/queries/clientpositive/temp_table_precedence.q
          A    ql/src/test/queries/clientpositive/authorization_create_temp_table.q
          A    ql/src/test/queries/clientpositive/temp_table_join1.q
          A    ql/src/test/results/clientnegative/temp_table_rename.q.out
          A    ql/src/test/results/clientnegative/temp_table_create_like_partitions.q.out
          A    ql/src/test/results/clientnegative/temp_table_partitions.q.out
          A    ql/src/test/results/clientnegative/temp_table_index.q.out
          A    ql/src/test/results/clientnegative/temp_table_column_stats.q.out
          A    ql/src/test/results/clientnegative/temp_table_authorize_create_tbl.q.out
          A    ql/src/test/results/clientpositive/temp_table_options1.q.out
          A    ql/src/test/results/clientpositive/temp_table_windowing_expressions.q.out
          A    ql/src/test/results/clientpositive/show_create_table_temp_table.q.out
          U    ql/src/test/results/clientpositive/nullformatCTAS.q.out
          A    ql/src/test/results/clientpositive/temp_table_external.q.out
          A    ql/src/test/results/clientpositive/temp_table_precedence.q.out
          U    ql/src/test/results/clientpositive/nullformat.q.out
          U    ql/src/test/results/clientpositive/show_create_table_delimited.q.out
          A    ql/src/test/results/clientpositive/temp_table_names.q.out
          A    ql/src/test/results/clientpositive/temp_table_subquery1.q.out
          A    ql/src/test/results/clientpositive/temp_table_gb1.q.out
          A    ql/src/test/results/clientpositive/authorization_create_temp_table.q.out
          A    ql/src/test/results/clientpositive/temp_table_join1.q.out
          U    ql/src/test/results/clientpositive/show_create_table_alter.q.out
          U    ql/src/test/results/clientpositive/show_create_table_db_table.q.out
          A    ql/src/test/results/clientpositive/tez/temp_table.q.out
          A    ql/src/test/results/clientpositive/temp_table.q.out
          U    ql/src/test/results/clientpositive/show_create_table_serde.q.out
          U    ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezTask.java
          U    ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager.java
          U    ql/src/test/org/apache/hadoop/hive/ql/parse/authorization/TestHiveAuthorizationTaskFactory.java
          U    ql/src/test/org/apache/hadoop/hive/ql/parse/TestMacroSemanticAnalyzer.java
          A    ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java
          U    ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java
          U    ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
          U    ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
          U    ql/src/java/org/apache/hadoop/hive/ql/Context.java
          U    ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g
          U    ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java
          U    ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
          U    ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java
          U    ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java
          U    ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java
          U    ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableLikeDesc.java
          
          Fetching external item into 'hcatalog/src/test/e2e/harness'
          Updated external to revision 1609306.
          
          Updated to revision 1609305.
          + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
          + patchFilePath=/data/hive-ptest/working/scratch/build.patch
          + [[ -f /data/hive-ptest/working/scratch/build.patch ]]
          + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
          + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch
          The patch does not appear to apply with p0, p1, or p2
          + exit 1
          '
          

          This message is automatically generated.

          ATTACHMENT ID: 12654861

          Show
          Hive QA added a comment - Overall : -1 no tests executed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12654861/HIVE-5976.5.patch Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/724/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/724/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-724/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]] + export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera + JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera + export PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin + PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + cd /data/hive-ptest/working/ + tee /data/hive-ptest/logs/PreCommit-HIVE-Build-724/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ svn = \s\v\n ]] + [[ -n '' ]] + [[ -d apache-svn-trunk-source ]] + [[ ! -d apache-svn-trunk-source/.svn ]] + [[ ! -d apache-svn-trunk-source ]] + cd apache-svn-trunk-source + svn revert -R . Reverted 'metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java' Reverted 'metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java' Reverted 'metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java' Reverted 'metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java' Reverted 'metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java' Reverted 'metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java' Reverted 'metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java' Reverted 'metastore/src/gen/thrift/gen-py/hive_metastore/ttypes.py' Reverted 'metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py' Reverted 'metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote' Reverted 'metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp' Reverted 'metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp' Reverted 'metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.h' Reverted 'metastore/src/gen/thrift/gen-cpp/hive_metastore_types.h' Reverted 'metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore_server.skeleton.cpp' Reverted 'metastore/src/gen/thrift/gen-rb/thrift_hive_metastore.rb' Reverted 'metastore/src/gen/thrift/gen-rb/hive_metastore_types.rb' Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PrivilegeBag.java' Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ColumnStatistics.java' Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Partition.java' Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/HiveObjectRef.java' Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionsStatsRequest.java' Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPrincipalsInRoleResponse.java' Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SkewedInfo.java' Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ShowCompactResponse.java' Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/RequestPartsSpec.java' Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AddPartitionsRequest.java' Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Type.java' Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/HeartbeatTxnRangeResponse.java' Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ShowLocksResponse.java' Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetOpenTxnsResponse.java' Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/OpenTxnsResponse.java' Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Schema.java' Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java' Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PrincipalPrivilegeSet.java' Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AddPartitionsResult.java' Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/TableStatsResult.java' Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/TableStatsRequest.java' Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionsByExprResult.java' Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Table.java' Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/LockRequest.java' Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/StorageDescriptor.java' Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetRoleGrantsForPrincipalResponse.java' Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetOpenTxnsInfoResponse.java' Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionsStatsResult.java' Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/DropPartitionsResult.java' Reverted 'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Function.java' Reverted 'metastore/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php' Reverted 'metastore/src/gen/thrift/gen-php/metastore/Types.php' Reverted 'metastore/if/hive_metastore.thrift' Reverted 'itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestAuthorizationApiAuthorizer.java' Reverted 'ql/src/test/results/clientpositive/authorization_role_grant1.q.out' Reverted 'ql/src/test/queries/clientpositive/authorization_role_grant1.q' Reverted 'ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g' Reverted 'ql/src/java/org/apache/hadoop/hive/ql/parse/authorization/HiveAuthorizationTaskFactoryImpl.java' Reverted 'ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java' Reverted 'ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java' Reverted 'ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/sqlstd/SQLStdHiveAccessController.java' ++ egrep -v '^X|^Performing status on external' ++ awk '{print $2}' ++ svn status --no-ignore + rm -rf target datanucleus.log ant/target shims/target shims/0.20/target shims/0.20S/target shims/0.23/target shims/aggregator/target shims/common/target shims/common-secure/target packaging/target hbase-handler/target testutils/target jdbc/target metastore/target metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GrantRevokeRoleRequest.java metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GrantRevokeRoleResponse.java metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GrantRevokeType.java itests/target itests/hcatalog-unit/target itests/test-serde/target itests/qtest/target itests/hive-unit-hadoop2/target itests/hive-minikdc/target itests/hive-unit/target itests/custom-serde/target itests/util/target hcatalog/target hcatalog/core/target hcatalog/streaming/target hcatalog/server-extensions/target hcatalog/hcatalog-pig-adapter/target hcatalog/webhcat/svr/target hcatalog/webhcat/java-client/target hwi/target common/target common/src/gen contrib/target service/target serde/target beeline/target odbc/target cli/target ql/dependency-reduced-pom.xml ql/target ql/src/test/results/clientnegative/authorization_role_grant2.q.out ql/src/test/queries/clientnegative/authorization_role_grant2.q + svn update U itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniMr.java U itests/qtest/testconfiguration.properties U metastore/if/hive_metastore.thrift U metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp U metastore/src/gen/thrift/gen-cpp/hive_metastore_types.h U metastore/src/gen/thrift/gen-rb/hive_metastore_types.rb U metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Table.java U metastore/src/gen/thrift/gen-php/metastore/Types.php U metastore/src/gen/thrift/gen-py/hive_metastore/ttypes.py U metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java U metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java A ql/src/test/queries/clientnegative/temp_table_rename.q A ql/src/test/queries/clientnegative/temp_table_index.q A ql/src/test/queries/clientnegative/temp_table_column_stats.q A ql/src/test/queries/clientnegative/temp_table_create_like_partitions.q A ql/src/test/queries/clientnegative/temp_table_authorize_create_tbl.q A ql/src/test/queries/clientnegative/temp_table_partitions.q A ql/src/test/queries/clientpositive/show_create_table_temp_table.q U ql/src/test/queries/clientpositive/stats19.q A ql/src/test/queries/clientpositive/temp_table_gb1.q A ql/src/test/queries/clientpositive/temp_table_subquery1.q A ql/src/test/queries/clientpositive/temp_table_external.q A ql/src/test/queries/clientpositive/temp_table_names.q A ql/src/test/queries/clientpositive/temp_table_options1.q A ql/src/test/queries/clientpositive/temp_table.q A ql/src/test/queries/clientpositive/temp_table_windowing_expressions.q A ql/src/test/queries/clientpositive/temp_table_precedence.q A ql/src/test/queries/clientpositive/authorization_create_temp_table.q A ql/src/test/queries/clientpositive/temp_table_join1.q A ql/src/test/results/clientnegative/temp_table_rename.q.out A ql/src/test/results/clientnegative/temp_table_create_like_partitions.q.out A ql/src/test/results/clientnegative/temp_table_partitions.q.out A ql/src/test/results/clientnegative/temp_table_index.q.out A ql/src/test/results/clientnegative/temp_table_column_stats.q.out A ql/src/test/results/clientnegative/temp_table_authorize_create_tbl.q.out A ql/src/test/results/clientpositive/temp_table_options1.q.out A ql/src/test/results/clientpositive/temp_table_windowing_expressions.q.out A ql/src/test/results/clientpositive/show_create_table_temp_table.q.out U ql/src/test/results/clientpositive/nullformatCTAS.q.out A ql/src/test/results/clientpositive/temp_table_external.q.out A ql/src/test/results/clientpositive/temp_table_precedence.q.out U ql/src/test/results/clientpositive/nullformat.q.out U ql/src/test/results/clientpositive/show_create_table_delimited.q.out A ql/src/test/results/clientpositive/temp_table_names.q.out A ql/src/test/results/clientpositive/temp_table_subquery1.q.out A ql/src/test/results/clientpositive/temp_table_gb1.q.out A ql/src/test/results/clientpositive/authorization_create_temp_table.q.out A ql/src/test/results/clientpositive/temp_table_join1.q.out U ql/src/test/results/clientpositive/show_create_table_alter.q.out U ql/src/test/results/clientpositive/show_create_table_db_table.q.out A ql/src/test/results/clientpositive/tez/temp_table.q.out A ql/src/test/results/clientpositive/temp_table.q.out U ql/src/test/results/clientpositive/show_create_table_serde.q.out U ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezTask.java U ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager.java U ql/src/test/org/apache/hadoop/hive/ql/parse/authorization/TestHiveAuthorizationTaskFactory.java U ql/src/test/org/apache/hadoop/hive/ql/parse/TestMacroSemanticAnalyzer.java A ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java U ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java U ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java U ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java U ql/src/java/org/apache/hadoop/hive/ql/Context.java U ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g U ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java U ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java U ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java U ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java U ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java U ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableLikeDesc.java Fetching external item into 'hcatalog/src/test/e2e/harness' Updated external to revision 1609306. Updated to revision 1609305. + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hive-ptest/working/scratch/build.patch + [[ -f /data/hive-ptest/working/scratch/build.patch ]] + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch The patch does not appear to apply with p0, p1, or p2 + exit 1 ' This message is automatically generated. ATTACHMENT ID: 12654861
          Hide
          David Chen added a comment -

          Previous patch was not created correctly. Attaching a new one.

          Show
          David Chen added a comment - Previous patch was not created correctly. Attaching a new one.
          Hide
          Hive QA added a comment -

          Overall: -1 no tests executed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12654880/HIVE-5976.5.patch

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/726/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/726/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-726/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Tests exited with: NonZeroExitCodeException
          Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
          + export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
          + JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
          + export PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
          + PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
          + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
          + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
          + export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
          + M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
          + cd /data/hive-ptest/working/
          + tee /data/hive-ptest/logs/PreCommit-HIVE-Build-726/source-prep.txt
          + [[ false == \t\r\u\e ]]
          + mkdir -p maven ivy
          + [[ svn = \s\v\n ]]
          + [[ -n '' ]]
          + [[ -d apache-svn-trunk-source ]]
          + [[ ! -d apache-svn-trunk-source/.svn ]]
          + [[ ! -d apache-svn-trunk-source ]]
          + cd apache-svn-trunk-source
          + svn revert -R .
          ++ egrep -v '^X|^Performing status on external'
          ++ awk '{print $2}'
          ++ svn status --no-ignore
          + rm -rf
          + svn update
          
          Fetching external item into 'hcatalog/src/test/e2e/harness'
          External at revision 1609331.
          
          At revision 1609331.
          + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
          + patchFilePath=/data/hive-ptest/working/scratch/build.patch
          + [[ -f /data/hive-ptest/working/scratch/build.patch ]]
          + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
          + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch
          The patch does not appear to apply with p0, p1, or p2
          + exit 1
          '
          

          This message is automatically generated.

          ATTACHMENT ID: 12654880

          Show
          Hive QA added a comment - Overall : -1 no tests executed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12654880/HIVE-5976.5.patch Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/726/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/726/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-726/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]] + export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera + JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera + export PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin + PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + cd /data/hive-ptest/working/ + tee /data/hive-ptest/logs/PreCommit-HIVE-Build-726/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ svn = \s\v\n ]] + [[ -n '' ]] + [[ -d apache-svn-trunk-source ]] + [[ ! -d apache-svn-trunk-source/.svn ]] + [[ ! -d apache-svn-trunk-source ]] + cd apache-svn-trunk-source + svn revert -R . ++ egrep -v '^X|^Performing status on external' ++ awk '{print $2}' ++ svn status --no-ignore + rm -rf + svn update Fetching external item into 'hcatalog/src/test/e2e/harness' External at revision 1609331. At revision 1609331. + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hive-ptest/working/scratch/build.patch + [[ -f /data/hive-ptest/working/scratch/build.patch ]] + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch The patch does not appear to apply with p0, p1, or p2 + exit 1 ' This message is automatically generated. ATTACHMENT ID: 12654880
          Hide
          David Chen added a comment -

          Let's try not using --no-prefix.

          Show
          David Chen added a comment - Let's try not using --no-prefix.
          Hide
          Hive QA added a comment -

          Overall: -1 no tests executed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12654885/HIVE-5976.5.patch

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/727/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/727/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-727/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Tests exited with: NonZeroExitCodeException
          Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
          + export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
          + JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
          + export PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
          + PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
          + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
          + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
          + export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
          + M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
          + cd /data/hive-ptest/working/
          + tee /data/hive-ptest/logs/PreCommit-HIVE-Build-727/source-prep.txt
          + [[ false == \t\r\u\e ]]
          + mkdir -p maven ivy
          + [[ svn = \s\v\n ]]
          + [[ -n '' ]]
          + [[ -d apache-svn-trunk-source ]]
          + [[ ! -d apache-svn-trunk-source/.svn ]]
          + [[ ! -d apache-svn-trunk-source ]]
          + cd apache-svn-trunk-source
          + svn revert -R .
          ++ awk '{print $2}'
          ++ egrep -v '^X|^Performing status on external'
          ++ svn status --no-ignore
          + rm -rf
          + svn update
          
          Fetching external item into 'hcatalog/src/test/e2e/harness'
          External at revision 1609333.
          
          At revision 1609333.
          + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
          + patchFilePath=/data/hive-ptest/working/scratch/build.patch
          + [[ -f /data/hive-ptest/working/scratch/build.patch ]]
          + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
          + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch
          The patch does not appear to apply with p0, p1, or p2
          + exit 1
          '
          

          This message is automatically generated.

          ATTACHMENT ID: 12654885

          Show
          Hive QA added a comment - Overall : -1 no tests executed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12654885/HIVE-5976.5.patch Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/727/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/727/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-727/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]] + export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera + JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera + export PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin + PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + cd /data/hive-ptest/working/ + tee /data/hive-ptest/logs/PreCommit-HIVE-Build-727/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ svn = \s\v\n ]] + [[ -n '' ]] + [[ -d apache-svn-trunk-source ]] + [[ ! -d apache-svn-trunk-source/.svn ]] + [[ ! -d apache-svn-trunk-source ]] + cd apache-svn-trunk-source + svn revert -R . ++ awk '{print $2}' ++ egrep -v '^X|^Performing status on external' ++ svn status --no-ignore + rm -rf + svn update Fetching external item into 'hcatalog/src/test/e2e/harness' External at revision 1609333. At revision 1609333. + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hive-ptest/working/scratch/build.patch + [[ -f /data/hive-ptest/working/scratch/build.patch ]] + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch The patch does not appear to apply with p0, p1, or p2 + exit 1 ' This message is automatically generated. ATTACHMENT ID: 12654885
          Hide
          David Chen added a comment -

          Ok, I tried uploading a patch using:

          • git format-patch trunk
          • git diff trunk --no-prefix
          • git diff trunk

          I am getting the The patch does not appear to apply with p0, p1, or p2 error for each of them.

          Is there something wrong with the Hive pre-commit test job?

          Show
          David Chen added a comment - Ok, I tried uploading a patch using: git format-patch trunk git diff trunk --no-prefix git diff trunk I am getting the The patch does not appear to apply with p0, p1, or p2 error for each of them. Is there something wrong with the Hive pre-commit test job?
          Hide
          Brock Noland added a comment -

          Can you verify your patch applies on top of trunk HEAD?

          Show
          Brock Noland added a comment - Can you verify your patch applies on top of trunk HEAD?
          Hide
          David Chen added a comment -

          Ah. I rebased before I submitted the first of those patches and thought the error message implied something wrong with the format of the patch itself.

          I have rebased on trunk again and resolved the merge conflict. Uploading a new patch.

          Show
          David Chen added a comment - Ah. I rebased before I submitted the first of those patches and thought the error message implied something wrong with the format of the patch itself. I have rebased on trunk again and resolved the merge conflict. Uploading a new patch.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12654895/HIVE-5976.6.patch

          ERROR: -1 due to 23 failed/errored test(s), 5720 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_8
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_union_table
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ctas
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ctas_colname
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ctas_uses_database_location
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_duplicate_key
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input15
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_inputddl1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_inputddl2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_inputddl3
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_merge3
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nonmr_fetch
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nullformat
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nullformatCTAS
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoin_noskew
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin9
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_storage_format_descriptor
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_temp_table
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union25
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_top_level
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_ctas
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_dml
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_parallel_orderby
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/728/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/728/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-728/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 23 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12654895

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12654895/HIVE-5976.6.patch ERROR: -1 due to 23 failed/errored test(s), 5720 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_union_table org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ctas org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ctas_colname org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ctas_uses_database_location org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_duplicate_key org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input15 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_inputddl1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_inputddl2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_inputddl3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_merge3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nonmr_fetch org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nullformat org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nullformatCTAS org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoin_noskew org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin9 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_storage_format_descriptor org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_temp_table org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union25 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_top_level org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_ctas org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_dml org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_parallel_orderby Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/728/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/728/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-728/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 23 tests failed This message is automatically generated. ATTACHMENT ID: 12654895
          Hide
          David Chen added a comment -

          I am looking into the test failures. It looks like my patch caused additional tests to fail due to an extra line in the output:

          serde name: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
          

          There are three test failures that appear to be caused by other problems.

          Finally, it seems that the test failures in the previous runs where the data size was different are no longer appearing.

          Show
          David Chen added a comment - I am looking into the test failures. It looks like my patch caused additional tests to fail due to an extra line in the output: serde name: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe There are three test failures that appear to be caused by other problems. Finally, it seems that the test failures in the previous runs where the data size was different are no longer appearing.
          Hide
          David Chen added a comment -

          Brock Noland I realized that the reason why the alter table tests are now passing and a number of the create table tests are now failing is that the create table and alter table codepaths do things differently in terms of setting the SerDe for text and sequencefile. The create table codepath does not set the SerDe for these two storage formats (see BaseSemanticAnalyzer. StorageFormat.fillStorageFormat()).

          However, the alter table codepath, in fact, does (see DDLSemanticAnalyze.analyzeAlterTableFileFormat()) and sets them to LazySimpleSerDe. However, now that both code paths go through the new StorageFormat class, not setting the SerDe to LazySimpleSerDe for text and sequencefile causes alter table fileformat to fail because the SerDe remains unchanged – which is clearly incorrect – but setting the SerDe to LazySimpleSerDe causes the create table tests to fail because now we have one extra line in the output because the SerDe is now being set.

          It seems to me that the create table codepath should also set the SerDe to LazySimpleSerDe for text and sequencefile. Is there a reason why it is currently not doing so?

          Show
          David Chen added a comment - Brock Noland I realized that the reason why the alter table tests are now passing and a number of the create table tests are now failing is that the create table and alter table codepaths do things differently in terms of setting the SerDe for text and sequencefile. The create table codepath does not set the SerDe for these two storage formats (see BaseSemanticAnalyzer. StorageFormat.fillStorageFormat()). However, the alter table codepath, in fact, does (see DDLSemanticAnalyze.analyzeAlterTableFileFormat()) and sets them to LazySimpleSerDe. However, now that both code paths go through the new StorageFormat class, not setting the SerDe to LazySimpleSerDe for text and sequencefile causes alter table fileformat to fail because the SerDe remains unchanged – which is clearly incorrect – but setting the SerDe to LazySimpleSerDe causes the create table tests to fail because now we have one extra line in the output because the SerDe is now being set. It seems to me that the create table codepath should also set the SerDe to LazySimpleSerDe for text and sequencefile. Is there a reason why it is currently not doing so?
          Hide
          Brock Noland added a comment -

          Thank you for your analysis!

          I think that both alter and create should set the serde. AFAIK it is populated in the HMS despite not being set. If tests fail because an additional line of output we'll just have to update them. Instructions on how to update them is under "How do I update the output of a CliDriver testcase?" here https://cwiki.apache.org/confluence/display/Hive/HiveDeveloperFAQ

          Show
          Brock Noland added a comment - Thank you for your analysis! I think that both alter and create should set the serde. AFAIK it is populated in the HMS despite not being set. If tests fail because an additional line of output we'll just have to update them. Instructions on how to update them is under "How do I update the output of a CliDriver testcase?" here https://cwiki.apache.org/confluence/display/Hive/HiveDeveloperFAQ
          Hide
          David Chen added a comment -

          Thanks, Brock Noland! I have updated the test output and attached a new patch.

          Show
          David Chen added a comment - Thanks, Brock Noland ! I have updated the test output and attached a new patch.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12655127/HIVE-5976.7.patch

          ERROR: -1 due to 3 failed/errored test(s), 5720 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_storage_format_descriptor
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_optimization
          org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/744/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/744/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-744/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 3 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12655127

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12655127/HIVE-5976.7.patch ERROR: -1 due to 3 failed/errored test(s), 5720 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_storage_format_descriptor org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_optimization org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/744/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/744/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-744/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 3 tests failed This message is automatically generated. ATTACHMENT ID: 12655127
          Hide
          Brock Noland added a comment -

          storage_format_descriptor might be related. Can you reproduce locally? The other two don't look to be.

          Show
          Brock Noland added a comment - storage_format_descriptor might be related. Can you reproduce locally? The other two don't look to be.
          Hide
          David Chen added a comment -

          I can reproduce it locally. It looks the failure is caused by additional lines in the output, specifically PREHOOK and POSTHOOK output printing the input file path (which is masked) and the table being written to.

          I think this does not appear to be an issue and should be fixed by updating the expected output.

          Show
          David Chen added a comment - I can reproduce it locally. It looks the failure is caused by additional lines in the output, specifically PREHOOK and POSTHOOK output printing the input file path (which is masked) and the table being written to. I think this does not appear to be an issue and should be fixed by updating the expected output.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12655358/HIVE-5976.8.patch

          ERROR: -1 due to 1 failed/errored test(s), 5706 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/754/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/754/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-754/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 1 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12655358

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12655358/HIVE-5976.8.patch ERROR: -1 due to 1 failed/errored test(s), 5706 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/754/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/754/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-754/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed This message is automatically generated. ATTACHMENT ID: 12655358
          Hide
          David Chen added a comment -

          This test failure does not appear to be related. I think this patch is ready to be committed unless there is additional feedback.

          Show
          David Chen added a comment - This test failure does not appear to be related. I think this patch is ready to be committed unless there is additional feedback.
          Hide
          Brock Noland added a comment -

          +1

          Show
          Brock Noland added a comment - +1
          Hide
          Brock Noland added a comment -

          David Chen I'd like to commit this today, but I committed HIVE-6037 first since it's such a huge patch which is PIA the rebase... Could you rebase this one?

          Show
          Brock Noland added a comment - David Chen I'd like to commit this today, but I committed HIVE-6037 first since it's such a huge patch which is PIA the rebase... Could you rebase this one?
          Hide
          David Chen added a comment -

          No problem. I have rebased against trunk and attached a new patch.

          Show
          David Chen added a comment - No problem. I have rebased against trunk and attached a new patch.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12655501/HIVE-5976.9.patch

          ERROR: -1 due to 3 failed/errored test(s), 5717 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_temp_table
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx
          org.apache.hive.hcatalog.cli.TestPermsGrp.testCustomPerms
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/775/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/775/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-775/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 3 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12655501

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12655501/HIVE-5976.9.patch ERROR: -1 due to 3 failed/errored test(s), 5717 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_temp_table org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx org.apache.hive.hcatalog.cli.TestPermsGrp.testCustomPerms Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/775/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/775/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-775/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 3 tests failed This message is automatically generated. ATTACHMENT ID: 12655501
          Hide
          Brock Noland added a comment -

          Thank you David for your contribution!! I have committed this to trunk!

          Show
          Brock Noland added a comment - Thank you David for your contribution!! I have committed this to trunk!
          Hide
          David Chen added a comment -

          Thanks, Brock!

          Show
          David Chen added a comment - Thanks, Brock!
          Hide
          Lefty Leverenz added a comment -

          This adds configuration parameter hive.default.serde with its description to the new, improved HiveConf.java. (Also to hive-default.xml.template, but isn't that redundant now that HIVE-6037 is committed?) So the Configuration Properties wiki needs to be updated.

          What other documentation does this need? Here are some candidates for revision:

          Show
          Lefty Leverenz added a comment - This adds configuration parameter hive.default.serde with its description to the new, improved HiveConf.java. (Also to hive-default.xml.template, but isn't that redundant now that HIVE-6037 is committed?) So the Configuration Properties wiki needs to be updated. What other documentation does this need? Here are some candidates for revision: SerDe Developer Guide – Hive SerDe DDL – CREATE TABLE syntax DDL – Create Table – Row Format, Storage Format, and SerDe DDL – Alter Table – Add SerDe Properties (maybe) DDL – CTAS (maybe) DDL – Alter Table/Partition File Format Hive Storage Handlers – DDL HCatalog Storage Formats (maybe) Avro SerDe (no change?) Parquet – HiveQL Syntax (no change?) ORC – HiveQL Syntax (maybe) Getting Started – Apache Weblog Data (no examples yet, but could add some) Tutorial – Usage and Examples
          Hide
          Thejas M Nair added a comment -

          This has been fixed in 0.14 release. Please open new jira if you see any issues.

          Show
          Thejas M Nair added a comment - This has been fixed in 0.14 release. Please open new jira if you see any issues.
          Hide
          Szehon Ho added a comment -

          It's mostly an internal implementation change, so I think none of the DDL pages have to change as there is no DDL syntax change.

          I did add a section describing this change to the developer guide for any developers to know about the registration mechanism:https://cwiki.apache.org/confluence/display/Hive/DeveloperGuide#DeveloperGuide-RegistrationofNativeSerDe's.

          Feel free to edit/move it if there is a better place, suggest more edits, or remove the TODOC label if this is sufficient.

          Show
          Szehon Ho added a comment - It's mostly an internal implementation change, so I think none of the DDL pages have to change as there is no DDL syntax change. I did add a section describing this change to the developer guide for any developers to know about the registration mechanism: https://cwiki.apache.org/confluence/display/Hive/DeveloperGuide#DeveloperGuide-RegistrationofNativeSerDe's . Feel free to edit/move it if there is a better place, suggest more edits, or remove the TODOC label if this is sufficient.
          Hide
          Szehon Ho added a comment -
          Show
          Szehon Ho added a comment - Also added property description 'hive.default.serde' to https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-FileFormatsandI/O
          Hide
          Lefty Leverenz added a comment -

          Nicely documented, Szehon Ho. Removing the TODOC14 label.

          Show
          Lefty Leverenz added a comment - Nicely documented, Szehon Ho . Removing the TODOC14 label.

            People

            • Assignee:
              David Chen
              Reporter:
              Brock Noland
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development