Hive
  1. Hive
  2. HIVE-6035

Windows: percentComplete returned by job status from WebHCat is null

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.12.0
    • Fix Version/s: 0.14.0
    • Component/s: HCatalog
    • Labels:
      None

      Description

      HIVE-5511 fixed the same problem on Linux, but it still broke on Windows.

      1. HIVE-6035.patch
        3 kB
        shanyu zhao
      2. HIVE-6035.1.patch
        3 kB
        shanyu zhao
      3. HIVE-6035.2.patch
        3 kB
        shanyu zhao
      4. HIVE-6035.3.patch
        3 kB
        Sushanth Sowmyan

        Issue Links

          Activity

          Hide
          shanyu zhao added a comment -

          The root cause of this problem is that the -Dlog4j.configuration needs to be in the format of "file://d:/xxx", so:
          "file://" + log4jProps.getAbsolutePath()
          works on Linux but does not work on Windows.

          This patch fixed two problem:
          1) templeton.cmd correctly set WEBHCAT_CONF_DIR so that override_container_log4j.properties can be copied to distributed cache. I also modified it so that one can run timepleton.cmd (without any parameter) in standalone mode.
          2) use hadoop class Path to convert the override log4j properties file path.

          Show
          shanyu zhao added a comment - The root cause of this problem is that the -Dlog4j.configuration needs to be in the format of "file://d:/xxx", so: "file://" + log4jProps.getAbsolutePath() works on Linux but does not work on Windows. This patch fixed two problem: 1) templeton.cmd correctly set WEBHCAT_CONF_DIR so that override_container_log4j.properties can be copied to distributed cache. I also modified it so that one can run timepleton.cmd (without any parameter) in standalone mode. 2) use hadoop class Path to convert the override log4j properties file path.
          Hide
          Hive QA added a comment -

          Overall: -1 no tests executed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12618722/HIVE-6035.patch

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/673/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/673/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Tests exited with: NonZeroExitCodeException
          Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n '' ]]
          + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
          + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
          + export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
          + M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
          + cd /data/hive-ptest/working/
          + tee /data/hive-ptest/logs/PreCommit-HIVE-Build-673/source-prep.txt
          + [[ false == \t\r\u\e ]]
          + mkdir -p maven ivy
          + [[ svn = \s\v\n ]]
          + [[ -n '' ]]
          + [[ -d apache-svn-trunk-source ]]
          + [[ ! -d apache-svn-trunk-source/.svn ]]
          + [[ ! -d apache-svn-trunk-source ]]
          + cd apache-svn-trunk-source
          + svn revert -R .
          Reverted 'jdbc/src/java/org/apache/hive/jdbc/Utils.java'
          ++ egrep -v '^X|^Performing status on external'
          ++ awk '{print $2}'
          ++ svn status --no-ignore
          + rm -rf target datanucleus.log ant/target shims/target shims/0.20/target shims/assembly/target shims/0.20S/target shims/0.23/target shims/common/target shims/common-secure/target packaging/target hbase-handler/target testutils/target jdbc/target metastore/target itests/target itests/hcatalog-unit/target itests/test-serde/target itests/qtest/target itests/hive-unit/target itests/custom-serde/target itests/util/target hcatalog/target hcatalog/storage-handlers/hbase/target hcatalog/server-extensions/target hcatalog/core/target hcatalog/webhcat/svr/target hcatalog/webhcat/java-client/target hcatalog/hcatalog-pig-adapter/target hwi/target common/target common/src/gen service/target contrib/target serde/target beeline/target odbc/target cli/target ql/dependency-reduced-pom.xml ql/target
          + svn update
          
          Fetching external item into 'hcatalog/src/test/e2e/harness'
          External at revision 1551750.
          
          At revision 1551750.
          + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
          + patchFilePath=/data/hive-ptest/working/scratch/build.patch
          + [[ -f /data/hive-ptest/working/scratch/build.patch ]]
          + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
          + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch
          The patch does not appear to apply with p0, p1, or p2
          + exit 1
          '
          

          This message is automatically generated.

          ATTACHMENT ID: 12618722

          Show
          Hive QA added a comment - Overall : -1 no tests executed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12618722/HIVE-6035.patch Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/673/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/673/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n '' ]] + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + cd /data/hive-ptest/working/ + tee /data/hive-ptest/logs/PreCommit-HIVE-Build-673/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ svn = \s\v\n ]] + [[ -n '' ]] + [[ -d apache-svn-trunk-source ]] + [[ ! -d apache-svn-trunk-source/.svn ]] + [[ ! -d apache-svn-trunk-source ]] + cd apache-svn-trunk-source + svn revert -R . Reverted 'jdbc/src/java/org/apache/hive/jdbc/Utils.java' ++ egrep -v '^X|^Performing status on external' ++ awk '{print $2}' ++ svn status --no-ignore + rm -rf target datanucleus.log ant/target shims/target shims/0.20/target shims/assembly/target shims/0.20S/target shims/0.23/target shims/common/target shims/common-secure/target packaging/target hbase-handler/target testutils/target jdbc/target metastore/target itests/target itests/hcatalog-unit/target itests/test-serde/target itests/qtest/target itests/hive-unit/target itests/custom-serde/target itests/util/target hcatalog/target hcatalog/storage-handlers/hbase/target hcatalog/server-extensions/target hcatalog/core/target hcatalog/webhcat/svr/target hcatalog/webhcat/java-client/target hcatalog/hcatalog-pig-adapter/target hwi/target common/target common/src/gen service/target contrib/target serde/target beeline/target odbc/target cli/target ql/dependency-reduced-pom.xml ql/target + svn update Fetching external item into 'hcatalog/src/test/e2e/harness' External at revision 1551750. At revision 1551750. + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hive-ptest/working/scratch/build.patch + [[ -f /data/hive-ptest/working/scratch/build.patch ]] + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch The patch does not appear to apply with p0, p1, or p2 + exit 1 ' This message is automatically generated. ATTACHMENT ID: 12618722
          Hide
          Eugene Koifman added a comment -

          +1

          Show
          Eugene Koifman added a comment - +1
          Hide
          shanyu zhao added a comment -

          attaching a new patch

          Show
          shanyu zhao added a comment - attaching a new patch
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12619912/HIVE-6035.1.patch

          ERROR: -1 due to 1 failed/errored test(s), 4811 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket_num_reducers
          

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/724/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/724/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 1 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12619912

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12619912/HIVE-6035.1.patch ERROR: -1 due to 1 failed/errored test(s), 4811 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket_num_reducers Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/724/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/724/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed This message is automatically generated. ATTACHMENT ID: 12619912
          Hide
          shanyu zhao added a comment -

          The failure has nothing to do with this patch. Also there's no unit tests for webhcat project. This change can be verified by webhcat e2e tests.

          Show
          shanyu zhao added a comment - The failure has nothing to do with this patch. Also there's no unit tests for webhcat project. This change can be verified by webhcat e2e tests.
          Hide
          shanyu zhao added a comment -

          updated templeton.cmd to work with both hadoop-1 and hadoop-2.

          Show
          shanyu zhao added a comment - updated templeton.cmd to work with both hadoop-1 and hadoop-2.
          Hide
          shanyu zhao added a comment -

          review board added

          Show
          shanyu zhao added a comment - review board added
          Hide
          Hive QA added a comment -

          Overall: +1 all checks pass

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12623767/HIVE-6035.2.patch

          SUCCESS: +1 4942 tests passed

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/954/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/954/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          

          This message is automatically generated.

          ATTACHMENT ID: 12623767

          Show
          Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12623767/HIVE-6035.2.patch SUCCESS: +1 4942 tests passed Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/954/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/954/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated. ATTACHMENT ID: 12623767
          Hide
          Sushanth Sowmyan added a comment -

          I'm going to cancel this patch, upload a new version of the same patch (so it applies cleanly on trunk) and re-mark it as patch available. If precommit tests do not have a problem, I'll go ahead and commit.

          Show
          Sushanth Sowmyan added a comment - I'm going to cancel this patch, upload a new version of the same patch (so it applies cleanly on trunk) and re-mark it as patch available. If precommit tests do not have a problem, I'll go ahead and commit.
          Hide
          Sushanth Sowmyan added a comment -

          Attached new patch. Also, please note that I have removed the fix version for this patch, since fix version is supposed to be set only after a commit has been made, and tracks what version the patch has been applied to, not which version the commit is desired in.

          Show
          Sushanth Sowmyan added a comment - Attached new patch. Also, please note that I have removed the fix version for this patch, since fix version is supposed to be set only after a commit has been made, and tracks what version the patch has been applied to, not which version the commit is desired in.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12639904/HIVE-6035.3.patch

          ERROR: -1 due to 2 failed/errored test(s), 5614 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_auto_sortmerge_join_16
          org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_minimr_broken_pipe
          

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2232/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2232/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 2 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12639904

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12639904/HIVE-6035.3.patch ERROR: -1 due to 2 failed/errored test(s), 5614 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_auto_sortmerge_join_16 org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_minimr_broken_pipe Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2232/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2232/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed This message is automatically generated. ATTACHMENT ID: 12639904
          Hide
          Sushanth Sowmyan added a comment -

          Fixed on 0.14 trunk. Thanks, Shanyu!

          Show
          Sushanth Sowmyan added a comment - Fixed on 0.14 trunk. Thanks, Shanyu!
          Hide
          Thejas M Nair added a comment -

          This has been fixed in 0.14 release. Please open new jira if you see any issues.

          Show
          Thejas M Nair added a comment - This has been fixed in 0.14 release. Please open new jira if you see any issues.

            People

            • Assignee:
              shanyu zhao
              Reporter:
              shanyu zhao
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development