Hive
  1. Hive
  2. HIVE-4723

DDLSemanticAnalyzer.addTablePartsOutputs eats several exceptions

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.12.0
    • Fix Version/s: 0.14.0
    • Component/s: None
    • Labels:
      None

      Description

      I accidently tried to archive a partition on a non-partitioned table. The error message was bad, hive ate an exception, and NPE'ed.

      2013-06-09 16:36:12,628 ERROR parse.DDLSemanticAnalyzer (DDLSemanticAnalyzer.java:addTablePartsOutputs(2899)) - Got HiveException during obtaining list of partitions
      2013-06-09 16:36:12,628 ERROR ql.Driver (SessionState.java:printError(383)) - FAILED: NullPointerException null
      java.lang.NullPointerException
      	at org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.addTablePartsOutputs(DDLSemanticAnalyzer.java:2912)
      	at org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.addTablePartsOutputs(DDLSemanticAnalyzer.java:2877)
      	at org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeAlterTableArchive(DDLSemanticAnalyzer.java:2730)
      	at org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeInternal(DDLSemanticAnalyzer.java:316)
      	at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:277)
      	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:433)
      	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:337)
      	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:902)
      	at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
      	at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
      	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
      	at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:782)
      	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
      	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:597)
      	at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
      
      1. HIVE-4723.patch
        4 kB
        Szehon Ho
      2. HIVE-4723.5.patch
        7 kB
        Szehon Ho
      3. HIVE-4723.5.patch
        7 kB
        Brock Noland
      4. HIVE-4723.4.patch
        7 kB
        Szehon Ho
      5. HIVE-4723.3.patch
        6 kB
        Szehon Ho
      6. HIVE-4723.2.patch
        6 kB
        Szehon Ho
      7. HIVE-4723.1.patch
        6 kB
        Szehon Ho

        Issue Links

          Activity

          Hide
          Ashutosh Chauhan added a comment -

          Committed to trunk. Thanks, Szehon!

          Show
          Ashutosh Chauhan added a comment - Committed to trunk. Thanks, Szehon!
          Hide
          Ashutosh Chauhan added a comment -

          +1

          Show
          Ashutosh Chauhan added a comment - +1
          Hide
          Hive QA added a comment -

          Overall: +1 all checks pass

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12614128/HIVE-4723.5.patch

          SUCCESS: +1 4614 tests passed

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/326/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/326/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          

          This message is automatically generated.

          ATTACHMENT ID: 12614128

          Show
          Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12614128/HIVE-4723.5.patch SUCCESS: +1 4614 tests passed Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/326/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/326/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated. ATTACHMENT ID: 12614128
          Hide
          Brock Noland added a comment -

          Re-uploading...my bad.

          Show
          Brock Noland added a comment - Re-uploading...my bad.
          Hide
          Hive QA added a comment -

          Overall: -1 no tests executed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12613970/HIVE-4723.5.patch

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/317/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/317/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Tests failed with: RuntimeException: java.lang.InterruptedException
          

          This message is automatically generated.

          ATTACHMENT ID: 12613970

          Show
          Hive QA added a comment - Overall : -1 no tests executed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12613970/HIVE-4723.5.patch Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/317/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/317/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Tests failed with: RuntimeException: java.lang.InterruptedException This message is automatically generated. ATTACHMENT ID: 12613970
          Hide
          Szehon Ho added a comment -

          Got the following error in the pre-commit tests:

          2013-11-14 18:05:07,142 WARN ExecutionPhase.replaceBadHosts:135 Only 0 hosts out of an expected 16, attempting to replace bad hosts
          2013-11-14 18:26:25,422 INFO ExecutionPhase.execute:113 PERF: exec phase 96 minutes
          2013-11-14 18:26:25,422 ERROR PTest.run:173 Test run exited with an unexpected error java.lang.IllegalArgumentException
          at java.util.concurrent.ThreadPoolExecutor.<init>(ThreadPoolExecutor.java:589)
          at java.util.concurrent.ThreadPoolExecutor.<init>(ThreadPoolExecutor.java:480)
          at java.util.concurrent.Executors.newFixedThreadPool(Executors.java:59)
          at org.apache.hive.ptest.execution.context.CloudExecutionContextProvider.verifyHosts(CloudExecutionContextProvider.java:245)
          at org.apache.hive.ptest.execution.context.CloudExecutionContextProvider.createNodes(CloudExecutionContextProvider.java:208)
          at org.apache.hive.ptest.execution.context.CloudExecutionContextProvider.replaceBadHosts(CloudExecutionContextProvider.java:167)
          at org.apache.hive.ptest.execution.context.ExecutionContext.replaceBadHosts(ExecutionContext.java:68)
          at org.apache.hive.ptest.execution.ExecutionPhase.replaceBadHosts(ExecutionPhase.java:138)
          at org.apache.hive.ptest.execution.ExecutionPhase.execute(ExecutionPhase.java:93)
          at org.apache.hive.ptest.execution.PTest.run(PTest.java:162)
          at org.apache.hive.ptest.api.server.TestExecutor.run(TestExecutor.java:118)

          I am guessing its a test-framework sporadic error nothing to do with this change, re-attaching new patch to try again.

          Show
          Szehon Ho added a comment - Got the following error in the pre-commit tests: 2013-11-14 18:05:07,142 WARN ExecutionPhase.replaceBadHosts:135 Only 0 hosts out of an expected 16, attempting to replace bad hosts 2013-11-14 18:26:25,422 INFO ExecutionPhase.execute:113 PERF: exec phase 96 minutes 2013-11-14 18:26:25,422 ERROR PTest.run:173 Test run exited with an unexpected error java.lang.IllegalArgumentException at java.util.concurrent.ThreadPoolExecutor.<init>(ThreadPoolExecutor.java:589) at java.util.concurrent.ThreadPoolExecutor.<init>(ThreadPoolExecutor.java:480) at java.util.concurrent.Executors.newFixedThreadPool(Executors.java:59) at org.apache.hive.ptest.execution.context.CloudExecutionContextProvider.verifyHosts(CloudExecutionContextProvider.java:245) at org.apache.hive.ptest.execution.context.CloudExecutionContextProvider.createNodes(CloudExecutionContextProvider.java:208) at org.apache.hive.ptest.execution.context.CloudExecutionContextProvider.replaceBadHosts(CloudExecutionContextProvider.java:167) at org.apache.hive.ptest.execution.context.ExecutionContext.replaceBadHosts(ExecutionContext.java:68) at org.apache.hive.ptest.execution.ExecutionPhase.replaceBadHosts(ExecutionPhase.java:138) at org.apache.hive.ptest.execution.ExecutionPhase.execute(ExecutionPhase.java:93) at org.apache.hive.ptest.execution.PTest.run(PTest.java:162) at org.apache.hive.ptest.api.server.TestExecutor.run(TestExecutor.java:118) I am guessing its a test-framework sporadic error nothing to do with this change, re-attaching new patch to try again.
          Hide
          Hive QA added a comment -

          Overall: -1 no tests executed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12613733/HIVE-4723.4.patch

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/284/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/284/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Tests failed with: IllegalArgumentException: null
          

          This message is automatically generated.

          ATTACHMENT ID: 12613733

          Show
          Hive QA added a comment - Overall : -1 no tests executed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12613733/HIVE-4723.4.patch Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/284/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/284/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Tests failed with: IllegalArgumentException: null This message is automatically generated. ATTACHMENT ID: 12613733
          Hide
          Szehon Ho added a comment -

          Looks like files got moved with mavenization. Generating a new patch for that.

          Show
          Szehon Ho added a comment - Looks like files got moved with mavenization. Generating a new patch for that.
          Hide
          Hive QA added a comment -

          Overall: -1 no tests executed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12613415/HIVE-4723.3.patch

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/254/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/254/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Tests failed with: NonZeroExitCodeException: Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n '' ]]
          + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
          + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
          + export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
          + M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
          + cd /data/hive-ptest/working/
          + tee /data/hive-ptest/logs/PreCommit-HIVE-Build-254/source-prep.txt
          + [[ false == \t\r\u\e ]]
          + mkdir -p maven ivy
          + [[ svn = \s\v\n ]]
          + [[ -n '' ]]
          + [[ -d apache-svn-trunk-source ]]
          + [[ ! -d apache-svn-trunk-source/.svn ]]
          + [[ ! -d apache-svn-trunk-source ]]
          + cd apache-svn-trunk-source
          + svn revert -R .
          ++ egrep -v '^X|^Performing status on external'
          ++ awk '{print $2}'
          ++ svn status --no-ignore
          + rm -rf target datanucleus.log ant/target shims/target shims/0.20/target shims/assembly/target shims/0.20S/target shims/0.23/target shims/common/target shims/common-secure/target packaging/target hbase-handler/target testutils/target jdbc/target metastore/target itests/target itests/hcatalog-unit/target itests/test-serde/target itests/qtest/target itests/hive-unit/target itests/custom-serde/target itests/util/target hcatalog/target hcatalog/storage-handlers/hbase/target hcatalog/server-extensions/target hcatalog/core/target hcatalog/webhcat/svr/target hcatalog/webhcat/java-client/target hcatalog/hcatalog-pig-adapter/target hwi/target common/target common/src/gen contrib/target service/target serde/target beeline/target odbc/target cli/target ql/dependency-reduced-pom.xml ql/target
          + svn update
          
          Fetching external item into 'hcatalog/src/test/e2e/harness'
          External at revision 1541401.
          
          At revision 1541401.
          + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
          + patchFilePath=/data/hive-ptest/working/scratch/build.patch
          + [[ -f /data/hive-ptest/working/scratch/build.patch ]]
          + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
          + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch
          The patch does not appear to apply with p0, p1, or p2
          + exit 1
          '
          

          This message is automatically generated.

          ATTACHMENT ID: 12613415

          Show
          Hive QA added a comment - Overall : -1 no tests executed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12613415/HIVE-4723.3.patch Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/254/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/254/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Tests failed with: NonZeroExitCodeException: Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n '' ]] + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + cd /data/hive-ptest/working/ + tee /data/hive-ptest/logs/PreCommit-HIVE-Build-254/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ svn = \s\v\n ]] + [[ -n '' ]] + [[ -d apache-svn-trunk-source ]] + [[ ! -d apache-svn-trunk-source/.svn ]] + [[ ! -d apache-svn-trunk-source ]] + cd apache-svn-trunk-source + svn revert -R . ++ egrep -v '^X|^Performing status on external' ++ awk '{print $2}' ++ svn status --no-ignore + rm -rf target datanucleus.log ant/target shims/target shims/0.20/target shims/assembly/target shims/0.20S/target shims/0.23/target shims/common/target shims/common-secure/target packaging/target hbase-handler/target testutils/target jdbc/target metastore/target itests/target itests/hcatalog-unit/target itests/test-serde/target itests/qtest/target itests/hive-unit/target itests/custom-serde/target itests/util/target hcatalog/target hcatalog/storage-handlers/hbase/target hcatalog/server-extensions/target hcatalog/core/target hcatalog/webhcat/svr/target hcatalog/webhcat/java-client/target hcatalog/hcatalog-pig-adapter/target hwi/target common/target common/src/gen contrib/target service/target serde/target beeline/target odbc/target cli/target ql/dependency-reduced-pom.xml ql/target + svn update Fetching external item into 'hcatalog/src/test/e2e/harness' External at revision 1541401. At revision 1541401. + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hive-ptest/working/scratch/build.patch + [[ -f /data/hive-ptest/working/scratch/build.patch ]] + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch The patch does not appear to apply with p0, p1, or p2 + exit 1 ' This message is automatically generated. ATTACHMENT ID: 12613415
          Hide
          Szehon Ho added a comment -

          Attaching the same patch again to trigger testing. I believe the pre-commit test of the previous patch got lost in the queue the build moved to bigtop.

          Show
          Szehon Ho added a comment - Attaching the same patch again to trigger testing. I believe the pre-commit test of the previous patch got lost in the queue the build moved to bigtop.
          Hide
          Szehon Ho added a comment -

          Adding the exception to the log as per review request. Verified by checking hive.log.

          Show
          Szehon Ho added a comment - Adding the exception to the log as per review request. Verified by checking hive.log.
          Hide
          Szehon Ho added a comment -

          There were two negative tests that expected the exception to be swallowed during semantic analysis, and for execution to continue and the DDL Task to fail. Changed the test output accordingly to only expect the exception during the semantic analysis and not proceed further.

          Show
          Szehon Ho added a comment - There were two negative tests that expected the exception to be swallowed during semantic analysis, and for execution to continue and the DDL Task to fail. Changed the test output accordingly to only expect the exception during the semantic analysis and not proceed further.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12610432/HIVE-4723.patch

          ERROR: -1 due to 2 failed/errored test(s), 4483 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_alter_rename_partition_failure3
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_touch2
          

          Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1255/testReport
          Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1255/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests failed with: TestsFailedException: 2 tests failed
          

          This message is automatically generated.

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12610432/HIVE-4723.patch ERROR: -1 due to 2 failed/errored test(s), 4483 tests executed Failed tests: org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_alter_rename_partition_failure3 org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_touch2 Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1255/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1255/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests failed with: TestsFailedException: 2 tests failed This message is automatically generated.

            People

            • Assignee:
              Szehon Ho
              Reporter:
              Brock Noland
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development