Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-16898

Validation of source file after distcp in repl load

    Details

    • Type: Bug
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.0.0
    • Fix Version/s: 3.0.0
    • Component/s: HiveServer2
    • Labels:
      None

      Description

      time between deciding the source and destination path for distcp to invoking of distcp can have a change of the source file, hence distcp might copy the wrong file to destination, hence we should an additional check on the checksum of the source file path after distcp finishes to make sure the path didnot change during the copy process. if it has take additional steps to delete the previous file on destination and copy the new source and repeat the same process as above till we copy the correct file.

      1. HIVE-16898.1.patch
        16 kB
        Daniel Dai
      2. HIVE-16898.2.patch
        17 kB
        Daniel Dai
      3. HIVE-16898.3.patch
        17 kB
        Daniel Dai
      4. HIVE-16898.4.patch
        17 kB
        Daniel Dai
      5. HIVE-16898.5.patch
        17 kB
        Daniel Dai
      6. HIVE-16898.6.patch
        17 kB
        Daniel Dai
      7. HIVE-16898.7.patch
        17 kB
        Daniel Dai

        Issue Links

          Activity

          Hide
          daijy Daniel Dai added a comment -

          Attach patch. Cannot find a way to write a unit test. Manually test it with debugger: setup a breakpoint right before copy, and drop table in another session.

          Show
          daijy Daniel Dai added a comment - Attach patch. Cannot find a way to write a unit test. Manually test it with debugger: setup a breakpoint right before copy, and drop table in another session.
          Hide
          anishek anishek added a comment -

          Daniel Dai can you please provide a pull request for the same.

          Show
          anishek anishek added a comment - Daniel Dai can you please provide a pull request for the same.
          Hide
          daijy Daniel Dai added a comment -

          Created a review board for it.

          Show
          daijy Daniel Dai added a comment - Created a review board for it.
          Hide
          daijy Daniel Dai added a comment -

          Addressing Anishek's review comments.

          Show
          daijy Daniel Dai added a comment - Addressing Anishek's review comments.
          Hide
          hiveqa Hive QA added a comment -

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12887786/HIVE-16898.3.patch

          ERROR: -1 due to no test(s) being added or modified.

          ERROR: -1 due to 18 failed/errored test(s), 11043 tests executed
          Failed tests:

          TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=230)
          TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=230)
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_view] (batchId=39)
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_mask_hash] (batchId=28)
          org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats] (batchId=156)
          org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1] (batchId=170)
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[drop_table_failure2] (batchId=89)
          org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=234)
          org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=215)
          org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01 (batchId=215)
          org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testIncrementalLoadFailAndRetry (batchId=218)
          org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testConnection (batchId=240)
          org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testIsValid (batchId=240)
          org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testIsValidNeg (batchId=240)
          org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testNegativeProxyAuth (batchId=240)
          org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testNegativeTokenAuth (batchId=240)
          org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testProxyAuth (batchId=240)
          org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testTokenAuth (batchId=240)
          

          Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6880/testReport
          Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6880/console
          Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6880/

          Messages:

          Executing org.apache.hive.ptest.execution.TestCheckPhase
          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 18 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12887786 - PreCommit-HIVE-Build

          Show
          hiveqa Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12887786/HIVE-16898.3.patch ERROR: -1 due to no test(s) being added or modified. ERROR: -1 due to 18 failed/errored test(s), 11043 tests executed Failed tests: TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=230) TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=230) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_view] (batchId=39) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_mask_hash] (batchId=28) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats] (batchId=156) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1] (batchId=170) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[drop_table_failure2] (batchId=89) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=234) org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=215) org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01 (batchId=215) org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testIncrementalLoadFailAndRetry (batchId=218) org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testConnection (batchId=240) org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testIsValid (batchId=240) org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testIsValidNeg (batchId=240) org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testNegativeProxyAuth (batchId=240) org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testNegativeTokenAuth (batchId=240) org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testProxyAuth (batchId=240) org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testTokenAuth (batchId=240) Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6880/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6880/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6880/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 18 tests failed This message is automatically generated. ATTACHMENT ID: 12887786 - PreCommit-HIVE-Build
          Hide
          hiveqa Hive QA added a comment -

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12887786/HIVE-16898.3.patch

          ERROR: -1 due to no test(s) being added or modified.

          ERROR: -1 due to 10 failed/errored test(s), 11043 tests executed
          Failed tests:

          TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=230)
          TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=230)
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_view] (batchId=39)
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_mask_hash] (batchId=28)
          org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats] (batchId=156)
          org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1] (batchId=170)
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[drop_table_failure2] (batchId=89)
          org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=215)
          org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01 (batchId=215)
          org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testIncrementalLoadFailAndRetry (batchId=218)
          

          Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6881/testReport
          Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6881/console
          Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6881/

          Messages:

          Executing org.apache.hive.ptest.execution.TestCheckPhase
          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 10 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12887786 - PreCommit-HIVE-Build

          Show
          hiveqa Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12887786/HIVE-16898.3.patch ERROR: -1 due to no test(s) being added or modified. ERROR: -1 due to 10 failed/errored test(s), 11043 tests executed Failed tests: TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=230) TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=230) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_view] (batchId=39) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_mask_hash] (batchId=28) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats] (batchId=156) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1] (batchId=170) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[drop_table_failure2] (batchId=89) org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=215) org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01 (batchId=215) org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testIncrementalLoadFailAndRetry (batchId=218) Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6881/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6881/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6881/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 10 tests failed This message is automatically generated. ATTACHMENT ID: 12887786 - PreCommit-HIVE-Build
          Hide
          daijy Daniel Dai added a comment -

          Discussed with Anishek, update the patch again.

          Show
          daijy Daniel Dai added a comment - Discussed with Anishek, update the patch again.
          Hide
          hiveqa Hive QA added a comment -

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12887961/HIVE-16898.4.patch

          ERROR: -1 due to build exiting with an error

          Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6895/testReport
          Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6895/console
          Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6895/

          Messages:

          Executing org.apache.hive.ptest.execution.TestCheckPhase
          Executing org.apache.hive.ptest.execution.PrepPhase
          Tests exited with: NonZeroExitCodeException
          Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N'
          2017-09-20 04:29:05.312
          + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
          + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
          + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
          + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
          + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
          + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
          + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
          + export 'MAVEN_OPTS=-Xmx1g '
          + MAVEN_OPTS='-Xmx1g '
          + cd /data/hiveptest/working/
          + tee /data/hiveptest/logs/PreCommit-HIVE-Build-6895/source-prep.txt
          + [[ false == \t\r\u\e ]]
          + mkdir -p maven ivy
          + [[ git = \s\v\n ]]
          + [[ git = \g\i\t ]]
          + [[ -z master ]]
          + [[ -d apache-github-source-source ]]
          + [[ ! -d apache-github-source-source/.git ]]
          + [[ ! -d apache-github-source-source ]]
          + date '+%Y-%m-%d %T.%3N'
          2017-09-20 04:29:05.315
          + cd apache-github-source-source
          + git fetch origin
          + git reset --hard HEAD
          HEAD is now at 5608300 HIVE-17537 Move Warehouse class to standalone metastore.  This closes #252.  (Alan Gates, reviewed by Zoltan Haindrich)
          + git clean -f -d
          Removing standalone-metastore/src/gen/org/
          + git checkout master
          Already on 'master'
          Your branch is up-to-date with 'origin/master'.
          + git reset --hard origin/master
          HEAD is now at 5608300 HIVE-17537 Move Warehouse class to standalone metastore.  This closes #252.  (Alan Gates, reviewed by Zoltan Haindrich)
          + git merge --ff-only origin/master
          Already up-to-date.
          + date '+%Y-%m-%d %T.%3N'
          2017-09-20 04:29:05.930
          + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
          + patchFilePath=/data/hiveptest/working/scratch/build.patch
          + [[ -f /data/hiveptest/working/scratch/build.patch ]]
          + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
          + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch
          error: a/metastore/src/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java: No such file or directory
          error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/ReplCopyTask.java: No such file or directory
          error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java: No such file or directory
          The patch does not appear to apply with p0, p1, or p2
          + exit 1
          '
          

          This message is automatically generated.

          ATTACHMENT ID: 12887961 - PreCommit-HIVE-Build

          Show
          hiveqa Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12887961/HIVE-16898.4.patch ERROR: -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6895/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6895/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6895/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N' 2017-09-20 04:29:05.312 + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]] + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'MAVEN_OPTS=-Xmx1g ' + MAVEN_OPTS='-Xmx1g ' + cd /data/hiveptest/working/ + tee /data/hiveptest/logs/PreCommit-HIVE-Build-6895/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + date '+%Y-%m-%d %T.%3N' 2017-09-20 04:29:05.315 + cd apache-github-source-source + git fetch origin + git reset --hard HEAD HEAD is now at 5608300 HIVE-17537 Move Warehouse class to standalone metastore. This closes #252. (Alan Gates, reviewed by Zoltan Haindrich) + git clean -f -d Removing standalone-metastore/src/gen/org/ + git checkout master Already on 'master' Your branch is up-to-date with 'origin/master'. + git reset --hard origin/master HEAD is now at 5608300 HIVE-17537 Move Warehouse class to standalone metastore. This closes #252. (Alan Gates, reviewed by Zoltan Haindrich) + git merge --ff-only origin/master Already up-to-date. + date '+%Y-%m-%d %T.%3N' 2017-09-20 04:29:05.930 + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hiveptest/working/scratch/build.patch + [[ -f /data/hiveptest/working/scratch/build.patch ]] + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch error: a/metastore/src/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java: No such file or directory error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/ReplCopyTask.java: No such file or directory error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java: No such file or directory The patch does not appear to apply with p0, p1, or p2 + exit 1 ' This message is automatically generated. ATTACHMENT ID: 12887961 - PreCommit-HIVE-Build
          Hide
          daijy Daniel Dai added a comment -

          Rebase with master.

          Show
          daijy Daniel Dai added a comment - Rebase with master.
          Hide
          hiveqa Hive QA added a comment -

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12888016/HIVE-16898.5.patch

          ERROR: -1 due to no test(s) being added or modified.

          ERROR: -1 due to 10 failed/errored test(s), 11049 tests executed
          Failed tests:

          TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=230)
          TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=230)
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_view] (batchId=39)
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_mask_hash] (batchId=28)
          org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats] (batchId=156)
          org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1] (batchId=170)
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[drop_table_failure2] (batchId=89)
          org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=234)
          org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] (batchId=234)
          org.apache.hive.hcatalog.pig.TestTextFileHCatStorer.testWriteDecimalXY (batchId=183)
          

          Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6908/testReport
          Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6908/console
          Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6908/

          Messages:

          Executing org.apache.hive.ptest.execution.TestCheckPhase
          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 10 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12888016 - PreCommit-HIVE-Build

          Show
          hiveqa Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12888016/HIVE-16898.5.patch ERROR: -1 due to no test(s) being added or modified. ERROR: -1 due to 10 failed/errored test(s), 11049 tests executed Failed tests: TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=230) TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=230) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_view] (batchId=39) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_mask_hash] (batchId=28) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats] (batchId=156) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1] (batchId=170) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[drop_table_failure2] (batchId=89) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=234) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] (batchId=234) org.apache.hive.hcatalog.pig.TestTextFileHCatStorer.testWriteDecimalXY (batchId=183) Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6908/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6908/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6908/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 10 tests failed This message is automatically generated. ATTACHMENT ID: 12888016 - PreCommit-HIVE-Build
          Hide
          hiveqa Hive QA added a comment -

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12888328/HIVE-16898.7.patch

          ERROR: -1 due to build exiting with an error

          Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6926/testReport
          Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6926/console
          Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6926/

          Messages:

          Executing org.apache.hive.ptest.execution.TestCheckPhase
          Executing org.apache.hive.ptest.execution.PrepPhase
          Tests exited with: NonZeroExitCodeException
          Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N'
          2017-09-21 19:21:12.083
          + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
          + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
          + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
          + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
          + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
          + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
          + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
          + export 'MAVEN_OPTS=-Xmx1g '
          + MAVEN_OPTS='-Xmx1g '
          + cd /data/hiveptest/working/
          + tee /data/hiveptest/logs/PreCommit-HIVE-Build-6926/source-prep.txt
          + [[ false == \t\r\u\e ]]
          + mkdir -p maven ivy
          + [[ git = \s\v\n ]]
          + [[ git = \g\i\t ]]
          + [[ -z master ]]
          + [[ -d apache-github-source-source ]]
          + [[ ! -d apache-github-source-source/.git ]]
          + [[ ! -d apache-github-source-source ]]
          + date '+%Y-%m-%d %T.%3N'
          2017-09-21 19:21:12.086
          + cd apache-github-source-source
          + git fetch origin
          + git reset --hard HEAD
          HEAD is now at afd0d9f HIVE-17472: Drop-partition for multi-level partition fails, if data does not exist. (Chris Drome, reviewed by Mithun Radhakrishnan)
          + git clean -f -d
          + git checkout master
          Already on 'master'
          Your branch is up-to-date with 'origin/master'.
          + git reset --hard origin/master
          HEAD is now at afd0d9f HIVE-17472: Drop-partition for multi-level partition fails, if data does not exist. (Chris Drome, reviewed by Mithun Radhakrishnan)
          + git merge --ff-only origin/master
          Already up-to-date.
          + date '+%Y-%m-%d %T.%3N'
          2017-09-21 19:21:12.627
          + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
          + patchFilePath=/data/hiveptest/working/scratch/build.patch
          + [[ -f /data/hiveptest/working/scratch/build.patch ]]
          + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
          + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch
          error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/ReplCopyTask.java: No such file or directory
          error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java: No such file or directory
          error: a/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java: No such file or directory
          The patch does not appear to apply with p0, p1, or p2
          + exit 1
          '
          

          This message is automatically generated.

          ATTACHMENT ID: 12888328 - PreCommit-HIVE-Build

          Show
          hiveqa Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12888328/HIVE-16898.7.patch ERROR: -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6926/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6926/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6926/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N' 2017-09-21 19:21:12.083 + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]] + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'MAVEN_OPTS=-Xmx1g ' + MAVEN_OPTS='-Xmx1g ' + cd /data/hiveptest/working/ + tee /data/hiveptest/logs/PreCommit-HIVE-Build-6926/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + date '+%Y-%m-%d %T.%3N' 2017-09-21 19:21:12.086 + cd apache-github-source-source + git fetch origin + git reset --hard HEAD HEAD is now at afd0d9f HIVE-17472: Drop-partition for multi-level partition fails, if data does not exist. (Chris Drome, reviewed by Mithun Radhakrishnan) + git clean -f -d + git checkout master Already on 'master' Your branch is up-to-date with 'origin/master'. + git reset --hard origin/master HEAD is now at afd0d9f HIVE-17472: Drop-partition for multi-level partition fails, if data does not exist. (Chris Drome, reviewed by Mithun Radhakrishnan) + git merge --ff-only origin/master Already up-to-date. + date '+%Y-%m-%d %T.%3N' 2017-09-21 19:21:12.627 + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hiveptest/working/scratch/build.patch + [[ -f /data/hiveptest/working/scratch/build.patch ]] + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/ReplCopyTask.java: No such file or directory error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java: No such file or directory error: a/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java: No such file or directory The patch does not appear to apply with p0, p1, or p2 + exit 1 ' This message is automatically generated. ATTACHMENT ID: 12888328 - PreCommit-HIVE-Build
          Hide
          hiveqa Hive QA added a comment -

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12888328/HIVE-16898.7.patch

          ERROR: -1 due to build exiting with an error

          Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6934/testReport
          Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6934/console
          Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6934/

          Messages:

          Executing org.apache.hive.ptest.execution.TestCheckPhase
          Executing org.apache.hive.ptest.execution.PrepPhase
          Tests exited with: NonZeroExitCodeException
          Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N'
          2017-09-22 03:19:44.041
          + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
          + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
          + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
          + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
          + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
          + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
          + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
          + export 'MAVEN_OPTS=-Xmx1g '
          + MAVEN_OPTS='-Xmx1g '
          + cd /data/hiveptest/working/
          + tee /data/hiveptest/logs/PreCommit-HIVE-Build-6934/source-prep.txt
          + [[ false == \t\r\u\e ]]
          + mkdir -p maven ivy
          + [[ git = \s\v\n ]]
          + [[ git = \g\i\t ]]
          + [[ -z master ]]
          + [[ -d apache-github-source-source ]]
          + [[ ! -d apache-github-source-source/.git ]]
          + [[ ! -d apache-github-source-source ]]
          + date '+%Y-%m-%d %T.%3N'
          2017-09-22 03:19:44.044
          + cd apache-github-source-source
          + git fetch origin
          + git reset --hard HEAD
          HEAD is now at 266c505 HIVE-17570 : Fix view deletion related test failures (create_view.q etc) (Tao Li via Thejas Nair)
          + git clean -f -d
          Removing itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/security/
          Removing metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java.orig
          Removing service/src/java/org/apache/hive/service/auth/HiveDelegationTokenManager.java
          Removing service/src/java/org/apache/hive/service/auth/HiveTokenStoreDelegationTokenSecretManager.java
          Removing standalone-metastore/src/gen/org/
          Removing standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/security/DBTokenStore.java
          Removing standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/security/ZooKeeperTokenStore.java
          + git checkout master
          Already on 'master'
          Your branch is up-to-date with 'origin/master'.
          + git reset --hard origin/master
          HEAD is now at 266c505 HIVE-17570 : Fix view deletion related test failures (create_view.q etc) (Tao Li via Thejas Nair)
          + git merge --ff-only origin/master
          Already up-to-date.
          + date '+%Y-%m-%d %T.%3N'
          2017-09-22 03:19:44.955
          + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
          + patchFilePath=/data/hiveptest/working/scratch/build.patch
          + [[ -f /data/hiveptest/working/scratch/build.patch ]]
          + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
          + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch
          error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/ReplCopyTask.java: No such file or directory
          error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java: No such file or directory
          error: a/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java: No such file or directory
          The patch does not appear to apply with p0, p1, or p2
          + exit 1
          '
          

          This message is automatically generated.

          ATTACHMENT ID: 12888328 - PreCommit-HIVE-Build

          Show
          hiveqa Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12888328/HIVE-16898.7.patch ERROR: -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6934/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6934/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6934/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N' 2017-09-22 03:19:44.041 + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]] + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'MAVEN_OPTS=-Xmx1g ' + MAVEN_OPTS='-Xmx1g ' + cd /data/hiveptest/working/ + tee /data/hiveptest/logs/PreCommit-HIVE-Build-6934/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + date '+%Y-%m-%d %T.%3N' 2017-09-22 03:19:44.044 + cd apache-github-source-source + git fetch origin + git reset --hard HEAD HEAD is now at 266c505 HIVE-17570 : Fix view deletion related test failures (create_view.q etc) (Tao Li via Thejas Nair) + git clean -f -d Removing itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/security/ Removing metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java.orig Removing service/src/java/org/apache/hive/service/auth/HiveDelegationTokenManager.java Removing service/src/java/org/apache/hive/service/auth/HiveTokenStoreDelegationTokenSecretManager.java Removing standalone-metastore/src/gen/org/ Removing standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/security/DBTokenStore.java Removing standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/security/ZooKeeperTokenStore.java + git checkout master Already on 'master' Your branch is up-to-date with 'origin/master'. + git reset --hard origin/master HEAD is now at 266c505 HIVE-17570 : Fix view deletion related test failures (create_view.q etc) (Tao Li via Thejas Nair) + git merge --ff-only origin/master Already up-to-date. + date '+%Y-%m-%d %T.%3N' 2017-09-22 03:19:44.955 + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hiveptest/working/scratch/build.patch + [[ -f /data/hiveptest/working/scratch/build.patch ]] + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/ReplCopyTask.java: No such file or directory error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java: No such file or directory error: a/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java: No such file or directory The patch does not appear to apply with p0, p1, or p2 + exit 1 ' This message is automatically generated. ATTACHMENT ID: 12888328 - PreCommit-HIVE-Build

            People

            • Assignee:
              daijy Daniel Dai
              Reporter:
              anishek anishek
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:

                Development