Hive
  1. Hive
  2. HIVE-7374

SHOW COMPACTIONS fail with remote metastore when there are no compations

    Details

      Description

      Prerequistes :
      1. Remote metastore
      2. No compactions

      In CLI after doing this :
      show compactions;
      Return error :

      FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. org.apache.thrift.transport.TTransportException
      

      In metatore logs :

      2014-07-09 17:54:10,537 ERROR [pool-3-thread-20]: server.TThreadPoolServer (TThreadPoolServer.java:run(213)) - Thrift error occurred during processing of message.
      org.apache.thrift.protocol.TProtocolException: Required field 'compacts' is unset! Struct:ShowCompactResponse(compacts:null)
              at org.apache.hadoop.hive.metastore.api.ShowCompactResponse.validate(ShowCompactResponse.java:310)
              at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$show_compact_result.validate(ThriftHiveMetastore.java)
              at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$show_compact_result$show_compact_resultStandardScheme.write(ThriftHiveMetastore.java)
              at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$show_compact_result$show_compact_resultStandardScheme.write(ThriftHiveMetastore.java)
              at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$show_compact_result.write(ThriftHiveMetastore.java)
              at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:53)
              at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:103)
              at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
              at java.lang.Thread.run(Thread.java:744)
      
      1. HIVE-7374.2.patch
        5 kB
        Damien Carol
      2. HIVE-7374.1.patch
        0.8 kB
        Damien Carol

        Issue Links

          Activity

          Hide
          Thejas M Nair added a comment -

          This has been fixed in 0.14 release. Please open new jira if you see any issues.

          Show
          Thejas M Nair added a comment - This has been fixed in 0.14 release. Please open new jira if you see any issues.
          Hide
          Navis added a comment -

          Committed to trunk. Thanks, Damien.

          Show
          Navis added a comment - Committed to trunk. Thanks, Damien.
          Hide
          Navis added a comment -

          +1

          Show
          Navis added a comment - +1
          Hide
          Damien Carol added a comment -

          Work done. Ready for review.

          Show
          Damien Carol added a comment - Work done. Ready for review.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12654955/HIVE-7374.2.patch

          ERROR: -1 due to 1 failed/errored test(s), 5718 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_optimization
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/735/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/735/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-735/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 1 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12654955

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12654955/HIVE-7374.2.patch ERROR: -1 due to 1 failed/errored test(s), 5718 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_optimization Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/735/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/735/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-735/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed This message is automatically generated. ATTACHMENT ID: 12654955
          Hide
          Damien Carol added a comment -

          I rebased patch and fixed errors in unit tests.
          Ready for review.

          Show
          Damien Carol added a comment - I rebased patch and fixed errors in unit tests. Ready for review.
          Hide
          Damien Carol added a comment -

          I looked into the test failures. Here my conclusions :

          Many tests rely on the fact that the metastore returns an object ShowCompactResponse with compacts property at null
          But this patch changed that because this throws errors in Thrift (this property is required).
          I made this change in unit tests :

          ShowCompactResponse rsp = txnHandler.showCompact(new ShowCompactRequest());
          Assert.assertEquals(0, rsp.getCompactsSize());
          

          Instead of :

          ShowCompactResponse rsp = txnHandler.showCompact(new ShowCompactRequest());
          Assert.assertNull(rsp.getCompacts());
          

          This fix these tests :

          • org.apache.hadoop.hive.metastore.txn.TestCompactionTxnHandler.testMarkCleaned
          • org.apache.hadoop.hive.ql.txn.compactor.TestCleaner.cleanupAfterMajorPartitionCompaction
          • org.apache.hadoop.hive.ql.txn.compactor.TestCleaner.cleanupAfterMajorPartitionCompactionNoBase
          • org.apache.hadoop.hive.ql.txn.compactor.TestCleaner.cleanupAfterMajorTableCompaction
          • org.apache.hadoop.hive.ql.txn.compactor.TestCleaner.cleanupAfterMinorPartitionCompaction
          • org.apache.hadoop.hive.ql.txn.compactor.TestCleaner.cleanupAfterMinorTableCompaction
          • org.apache.hadoop.hive.ql.txn.compactor.TestInitiator.noCompactOnManyDifferentPartitionAborts
          • org.apache.hadoop.hive.ql.txn.compactor.TestInitiator.noCompactTableDeltaPctNotHighEnough
          • org.apache.hadoop.hive.ql.txn.compactor.TestInitiator.noCompactTableNotEnoughDeltas
          • org.apache.hadoop.hive.ql.txn.compactor.TestInitiator.noCompactWhenNoCompactSet

          This test IS NOT due to this patch.

          • org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
          Show
          Damien Carol added a comment - I looked into the test failures. Here my conclusions : Many tests rely on the fact that the metastore returns an object ShowCompactResponse with compacts property at null But this patch changed that because this throws errors in Thrift (this property is required). I made this change in unit tests : ShowCompactResponse rsp = txnHandler.showCompact( new ShowCompactRequest()); Assert.assertEquals(0, rsp.getCompactsSize()); Instead of : ShowCompactResponse rsp = txnHandler.showCompact( new ShowCompactRequest()); Assert.assertNull(rsp.getCompacts()); This fix these tests : org.apache.hadoop.hive.metastore.txn.TestCompactionTxnHandler.testMarkCleaned org.apache.hadoop.hive.ql.txn.compactor.TestCleaner.cleanupAfterMajorPartitionCompaction org.apache.hadoop.hive.ql.txn.compactor.TestCleaner.cleanupAfterMajorPartitionCompactionNoBase org.apache.hadoop.hive.ql.txn.compactor.TestCleaner.cleanupAfterMajorTableCompaction org.apache.hadoop.hive.ql.txn.compactor.TestCleaner.cleanupAfterMinorPartitionCompaction org.apache.hadoop.hive.ql.txn.compactor.TestCleaner.cleanupAfterMinorTableCompaction org.apache.hadoop.hive.ql.txn.compactor.TestInitiator.noCompactOnManyDifferentPartitionAborts org.apache.hadoop.hive.ql.txn.compactor.TestInitiator.noCompactTableDeltaPctNotHighEnough org.apache.hadoop.hive.ql.txn.compactor.TestInitiator.noCompactTableNotEnoughDeltas org.apache.hadoop.hive.ql.txn.compactor.TestInitiator.noCompactWhenNoCompactSet This test IS NOT due to this patch. org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12654828/HIVE-7374.1.patch

          ERROR: -1 due to 11 failed/errored test(s), 5686 tests executed
          Failed tests:

          org.apache.hadoop.hive.metastore.txn.TestCompactionTxnHandler.testMarkCleaned
          org.apache.hadoop.hive.ql.txn.compactor.TestCleaner.cleanupAfterMajorPartitionCompaction
          org.apache.hadoop.hive.ql.txn.compactor.TestCleaner.cleanupAfterMajorPartitionCompactionNoBase
          org.apache.hadoop.hive.ql.txn.compactor.TestCleaner.cleanupAfterMajorTableCompaction
          org.apache.hadoop.hive.ql.txn.compactor.TestCleaner.cleanupAfterMinorPartitionCompaction
          org.apache.hadoop.hive.ql.txn.compactor.TestCleaner.cleanupAfterMinorTableCompaction
          org.apache.hadoop.hive.ql.txn.compactor.TestInitiator.noCompactOnManyDifferentPartitionAborts
          org.apache.hadoop.hive.ql.txn.compactor.TestInitiator.noCompactTableDeltaPctNotHighEnough
          org.apache.hadoop.hive.ql.txn.compactor.TestInitiator.noCompactTableNotEnoughDeltas
          org.apache.hadoop.hive.ql.txn.compactor.TestInitiator.noCompactWhenNoCompactSet
          org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/722/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/722/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-722/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 11 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12654828

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12654828/HIVE-7374.1.patch ERROR: -1 due to 11 failed/errored test(s), 5686 tests executed Failed tests: org.apache.hadoop.hive.metastore.txn.TestCompactionTxnHandler.testMarkCleaned org.apache.hadoop.hive.ql.txn.compactor.TestCleaner.cleanupAfterMajorPartitionCompaction org.apache.hadoop.hive.ql.txn.compactor.TestCleaner.cleanupAfterMajorPartitionCompactionNoBase org.apache.hadoop.hive.ql.txn.compactor.TestCleaner.cleanupAfterMajorTableCompaction org.apache.hadoop.hive.ql.txn.compactor.TestCleaner.cleanupAfterMinorPartitionCompaction org.apache.hadoop.hive.ql.txn.compactor.TestCleaner.cleanupAfterMinorTableCompaction org.apache.hadoop.hive.ql.txn.compactor.TestInitiator.noCompactOnManyDifferentPartitionAborts org.apache.hadoop.hive.ql.txn.compactor.TestInitiator.noCompactTableDeltaPctNotHighEnough org.apache.hadoop.hive.ql.txn.compactor.TestInitiator.noCompactTableNotEnoughDeltas org.apache.hadoop.hive.ql.txn.compactor.TestInitiator.noCompactWhenNoCompactSet org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/722/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/722/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-722/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 11 tests failed This message is automatically generated. ATTACHMENT ID: 12654828
          Hide
          Damien Carol added a comment -

          Ready for review.
          Anyone can assign this ticket to me?

          Show
          Damien Carol added a comment - Ready for review. Anyone can assign this ticket to me?
          Hide
          Damien Carol added a comment -

          The bug is simple. An object of type ShowCompactResponse is build without list of ShowCompactResponseElement.
          This throws errors in thrift part because it's a required field.
          The patch adds an empty list when the ShowCompactResponse object is instantiated.

          Show
          Damien Carol added a comment - The bug is simple. An object of type ShowCompactResponse is build without list of ShowCompactResponseElement. This throws errors in thrift part because it's a required field. The patch adds an empty list when the ShowCompactResponse object is instantiated.

            People

            • Assignee:
              Damien Carol
              Reporter:
              Damien Carol
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development