Hive
  1. Hive
  2. HIVE-5951

improve performance of adding partitions from client

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.13.0
    • Component/s: None
    • Labels:
      None

      Description

      Adding partitions to metastore is currently very inefficient. There are small things like, for !ifNotExists case, DDLSemanticAnalyzer gets the full partition object for every spec (which is a network call to metastore), and then discards it instantly; there's also general problem that too much processing is done on client side. DDLSA should analyze the query and make one call to metastore (or maybe a set of batched calls if there are too many partitions in the command), metastore should then figure out stuff and insert in batch.

      1. HIVE-5951.nogen.patch
        85 kB
        Sergey Shelukhin
      2. HIVE-5951.patch
        703 kB
        Sergey Shelukhin
      3. HIVE-5951.nogen.patch
        87 kB
        Sergey Shelukhin
      4. HIVE-5951.01.patch
        705 kB
        Sergey Shelukhin
      5. HIVE-5951.nogen.patch
        89 kB
        Sergey Shelukhin
      6. HIVE-5951.02.patch
        707 kB
        Sergey Shelukhin
      7. HIVE-5951.nogen.patch
        89 kB
        Sergey Shelukhin
      8. HIVE-5951.03.patch
        707 kB
        Sergey Shelukhin
      9. HIVE-5951.04.patch
        713 kB
        Sergey Shelukhin
      10. HIVE-5951.05.patch
        715 kB
        Sergey Shelukhin
      11. HIVE-5951.07.patch
        718 kB
        Sergey Shelukhin
      12. HIVE-5951.07.patch
        718 kB
        Sergey Shelukhin

        Activity

        Hide
        Sergey Shelukhin added a comment -

        preliminary patches. Some more query results will probably change, so not submitting yet, I will run overnight and update.

        Show
        Sergey Shelukhin added a comment - preliminary patches. Some more query results will probably change, so not submitting yet, I will run overnight and update.
        Hide
        Sergey Shelukhin added a comment -
        Show
        Sergey Shelukhin added a comment - RB at https://reviews.apache.org/r/16074/
        Hide
        Sergey Shelukhin added a comment -

        updated patch, I think tests should pass although for some tests I got different results on my and other machine

        Show
        Sergey Shelukhin added a comment - updated patch, I think tests should pass although for some tests I got different results on my and other machine
        Hide
        Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12617474/HIVE-5951.01.patch

        ERROR: -1 due to 8 failed/errored test(s), 4460 tests executed
        Failed tests:

        org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_addpart1
        org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_alter_view_failure4
        org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_archive5
        org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_default_partition_name
        org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_join28
        org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_join32
        org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_stats_partialscan_autogether
        org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_stats_partscan_norcfile
        

        Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/556/testReport
        Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/556/console

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 8 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12617474

        Show
        Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12617474/HIVE-5951.01.patch ERROR: -1 due to 8 failed/errored test(s), 4460 tests executed Failed tests: org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_addpart1 org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_alter_view_failure4 org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_archive5 org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_default_partition_name org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_join28 org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_join32 org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_stats_partialscan_autogether org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_stats_partscan_norcfile Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/556/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/556/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 8 tests failed This message is automatically generated. ATTACHMENT ID: 12617474
        Hide
        Ashutosh Chauhan added a comment -

        Does this handle the case of insert query which dynamically creates partition and load partitions in metastore at the end of MR job. That code-path is in MoveTask::execute() -> Hive.loadDynamicPartitions() ? That path is likely to be more relevant for this problem instead of from a ddl command someone trying to add 100s of partitions?

        Show
        Ashutosh Chauhan added a comment - Does this handle the case of insert query which dynamically creates partition and load partitions in metastore at the end of MR job. That code-path is in MoveTask::execute() -> Hive.loadDynamicPartitions() ? That path is likely to be more relevant for this problem instead of from a ddl command someone trying to add 100s of partitions?
        Hide
        Sergey Shelukhin added a comment -

        no, this is only for static partitions for now

        Show
        Sergey Shelukhin added a comment - no, this is only for static partitions for now
        Hide
        Sergey Shelukhin added a comment -

        I am looking at neg tests, forgot to run those

        Show
        Sergey Shelukhin added a comment - I am looking at neg tests, forgot to run those
        Hide
        Ashutosh Chauhan added a comment -

        It will be good to show partition spec in explain plan (unless its too complicated) instead of <multiple partition spec>. Also for input / output hook, that stuff is used for authorization, so it will be good not to loose that.

        Show
        Ashutosh Chauhan added a comment - It will be good to show partition spec in explain plan (unless its too complicated) instead of <multiple partition spec>. Also for input / output hook, that stuff is used for authorization, so it will be good not to loose that.
        Hide
        Sergey Shelukhin added a comment -

        the partitions gone from output are the ones that were not added (already exist). Should they also be present in output?

        Show
        Sergey Shelukhin added a comment - the partitions gone from output are the ones that were not added (already exist). Should they also be present in output?
        Hide
        Ashutosh Chauhan added a comment -

        In explain plan or hooks ?

        Show
        Ashutosh Chauhan added a comment - In explain plan or hooks ?
        Hide
        Sergey Shelukhin added a comment -

        that is about not showing in hooks. That is because the new code does not put the partitions that were duplicate into output... that is easy to fix, but I wonder why they are needed

        Show
        Sergey Shelukhin added a comment - that is about not showing in hooks. That is because the new code does not put the partitions that were duplicate into output... that is easy to fix, but I wonder why they are needed
        Hide
        Ashutosh Chauhan added a comment -

        If they are duplicate then its fine not to show them.

        Show
        Ashutosh Chauhan added a comment - If they are duplicate then its fine not to show them.
        Hide
        Hive QA added a comment -

        Overall: +1 all checks pass

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12617525/HIVE-5951.02.patch

        SUCCESS: +1 4460 tests passed

        Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/559/testReport
        Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/559/console

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        

        This message is automatically generated.

        ATTACHMENT ID: 12617525

        Show
        Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12617525/HIVE-5951.02.patch SUCCESS: +1 4460 tests passed Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/559/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/559/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated. ATTACHMENT ID: 12617525
        Hide
        Sergey Shelukhin added a comment -

        addressing comment in the jira, minor changes

        Show
        Sergey Shelukhin added a comment - addressing comment in the jira, minor changes
        Hide
        Hive QA added a comment -

        Overall: +1 all checks pass

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12617880/HIVE-5951.03.patch

        SUCCESS: +1 4761 tests passed

        Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/583/testReport
        Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/583/console

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        

        This message is automatically generated.

        ATTACHMENT ID: 12617880

        Show
        Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12617880/HIVE-5951.03.patch SUCCESS: +1 4761 tests passed Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/583/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/583/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated. ATTACHMENT ID: 12617880
        Hide
        Ashutosh Chauhan added a comment -

        Can you comment why we need to add new api in metastore thrift interface? This is a public api and there should be strong justification why existing api (add_partitions()) doesn't fit the requirement and new one is required.

        Show
        Ashutosh Chauhan added a comment - Can you comment why we need to add new api in metastore thrift interface? This is a public api and there should be strong justification why existing api (add_partitions()) doesn't fit the requirement and new one is required.
        Hide
        Sergey Shelukhin added a comment -

        New parameters are needed for the API, and they cannot be added in backward compatible manner.
        Overall, it's a good idea to have APIs in request-response pattern, and not with separate parameters due to this issue... let me file brainstorming jira for that.

        Show
        Sergey Shelukhin added a comment - New parameters are needed for the API, and they cannot be added in backward compatible manner. Overall, it's a good idea to have APIs in request-response pattern, and not with separate parameters due to this issue... let me file brainstorming jira for that.
        Hide
        Sergey Shelukhin added a comment -

        Filed HIVE-6061

        Show
        Sergey Shelukhin added a comment - Filed HIVE-6061
        Hide
        Sergey Shelukhin added a comment -

        ping?

        Show
        Sergey Shelukhin added a comment - ping?
        Hide
        Ashutosh Chauhan added a comment - - edited

        I thought perf gain would be because of doing one call on backend DBMS (probably using directsql). But looking at newly added method in ObjectStore, we are still adding partitions one by one in for loop. I think there will be quite a bit of perf to have by making single call on backend db.
        Apart from perf, for atomicity guarantees it also make sense to do one call.

        Show
        Ashutosh Chauhan added a comment - - edited I thought perf gain would be because of doing one call on backend DBMS (probably using directsql). But looking at newly added method in ObjectStore, we are still adding partitions one by one in for loop. I think there will be quite a bit of perf to have by making single call on backend db. Apart from perf, for atomicity guarantees it also make sense to do one call.
        Hide
        Sergey Shelukhin added a comment -

        For atomicity guarantees there are transactions
        We do the loop on the backend, and no longer do e.g. existence checks for each partitions separately from client.
        Anyway there's no such thing as inserting multiple things in one go, other than maybe inserting into separate table and then putting into main table. Also, it's going to be a mess given how much stuff partition can have.

        Show
        Sergey Shelukhin added a comment - For atomicity guarantees there are transactions We do the loop on the backend, and no longer do e.g. existence checks for each partitions separately from client. Anyway there's no such thing as inserting multiple things in one go, other than maybe inserting into separate table and then putting into main table. Also, it's going to be a mess given how much stuff partition can have.
        Hide
        Ashutosh Chauhan added a comment -

        Added few comments on RB. What I meant is to use pm.makePersistentAll() instead of pm.makePersistent() Take a look at RB comment.
        Also, did you do any testing to see how much perf gain this change is getting us?

        Show
        Ashutosh Chauhan added a comment - Added few comments on RB. What I meant is to use pm.makePersistentAll() instead of pm.makePersistent() Take a look at RB comment. Also, did you do any testing to see how much perf gain this change is getting us?
        Hide
        Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12622418/HIVE-5951.04.patch

        ERROR: -1 due to 7 failed/errored test(s), 4917 tests executed
        Failed tests:

        org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_alter_table_add_partition
        org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_alter_view_failure5
        org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_illegal_partition_type2
        org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_illegal_partition_type3
        org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_illegal_partition_type4
        org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_load_wrong_noof_part
        org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_truncate_partition_column
        

        Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/852/testReport
        Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/852/console

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 7 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12622418

        Show
        Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12622418/HIVE-5951.04.patch ERROR: -1 due to 7 failed/errored test(s), 4917 tests executed Failed tests: org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_alter_table_add_partition org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_alter_view_failure5 org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_illegal_partition_type2 org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_illegal_partition_type3 org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_illegal_partition_type4 org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_load_wrong_noof_part org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_truncate_partition_column Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/852/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/852/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 7 tests failed This message is automatically generated. ATTACHMENT ID: 12622418
        Hide
        Sergey Shelukhin added a comment -

        Simple test; on my machine, when adding 1000 partitions on an empty table via "alter table add partition - partition - ...", it went from 54.194 seconds to 15.877 seconds (with recent patch). Let me look at the tests...

        Show
        Sergey Shelukhin added a comment - Simple test; on my machine, when adding 1000 partitions on an empty table via "alter table add partition - partition - ...", it went from 54.194 seconds to 15.877 seconds (with recent patch). Let me look at the tests...
        Hide
        Sergey Shelukhin added a comment -

        exception texts changed in some places

        Show
        Sergey Shelukhin added a comment - exception texts changed in some places
        Hide
        Ashutosh Chauhan added a comment -

        +1

        Show
        Ashutosh Chauhan added a comment - +1
        Hide
        Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12622457/HIVE-5951.05.patch

        ERROR: -1 due to 7 failed/errored test(s), 4917 tests executed
        Failed tests:

        org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_alter_table_add_partition
        org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_alter_view_failure5
        org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_alter_view_failure7
        org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_analyze1
        org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_dyn_part1
        org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_touch2
        org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_truncate_partition_column
        

        Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/859/testReport
        Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/859/console

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 7 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12622457

        Show
        Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12622457/HIVE-5951.05.patch ERROR: -1 due to 7 failed/errored test(s), 4917 tests executed Failed tests: org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_alter_table_add_partition org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_alter_view_failure5 org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_alter_view_failure7 org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_analyze1 org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_dyn_part1 org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_touch2 org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_truncate_partition_column Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/859/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/859/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 7 tests failed This message is automatically generated. ATTACHMENT ID: 12622457
        Hide
        Sergey Shelukhin added a comment -

        Update some tests; rename method in thrift better (in other JIRA I am adding more methods with request-response pattern, probably a good idea to call them all ..._req rather than ...2).

        Show
        Sergey Shelukhin added a comment - Update some tests; rename method in thrift better (in other JIRA I am adding more methods with request-response pattern, probably a good idea to call them all ..._req rather than ...2).
        Hide
        Hive QA added a comment -

        Overall: -1 no tests executed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12622722/HIVE-5951.07.patch

        Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/894/testReport
        Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/894/console

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Tests exited with: NonZeroExitCodeException
        Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n '' ]]
        + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
        + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
        + export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
        + M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
        + cd /data/hive-ptest/working/
        + tee /data/hive-ptest/logs/PreCommit-HIVE-Build-894/source-prep.txt
        + [[ false == \t\r\u\e ]]
        + mkdir -p maven ivy
        + [[ svn = \s\v\n ]]
        + [[ -n '' ]]
        + [[ -d apache-svn-trunk-source ]]
        + [[ ! -d apache-svn-trunk-source/.svn ]]
        + [[ ! -d apache-svn-trunk-source ]]
        + cd apache-svn-trunk-source
        + svn revert -R .
        ++ egrep -v '^X|^Performing status on external'
        ++ awk '{print $2}'
        ++ svn status --no-ignore
        + rm -rf target datanucleus.log ant/target shims/target shims/0.20/target shims/0.20S/target shims/0.23/target shims/aggregator/target shims/common/target shims/common-secure/target packaging/target hbase-handler/target testutils/target jdbc/target metastore/target itests/target itests/hcatalog-unit/target itests/test-serde/target itests/qtest/target itests/hive-unit/target itests/custom-serde/target itests/util/target hcatalog/target hcatalog/storage-handlers/hbase/target hcatalog/server-extensions/target hcatalog/core/target hcatalog/webhcat/svr/target hcatalog/webhcat/java-client/target hcatalog/hcatalog-pig-adapter/target hwi/target common/target common/src/gen service/target contrib/target serde/target beeline/target odbc/target cli/target ql/dependency-reduced-pom.xml ql/target
        + svn update
        
        Fetching external item into 'hcatalog/src/test/e2e/harness'
        External at revision 1557954.
        
        At revision 1557954.
        + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
        + patchFilePath=/data/hive-ptest/working/scratch/build.patch
        + [[ -f /data/hive-ptest/working/scratch/build.patch ]]
        + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
        + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch
        The patch does not appear to apply with p0, p1, or p2
        + exit 1
        '
        

        This message is automatically generated.

        ATTACHMENT ID: 12622722

        Show
        Hive QA added a comment - Overall : -1 no tests executed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12622722/HIVE-5951.07.patch Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/894/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/894/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n '' ]] + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + cd /data/hive-ptest/working/ + tee /data/hive-ptest/logs/PreCommit-HIVE-Build-894/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ svn = \s\v\n ]] + [[ -n '' ]] + [[ -d apache-svn-trunk-source ]] + [[ ! -d apache-svn-trunk-source/.svn ]] + [[ ! -d apache-svn-trunk-source ]] + cd apache-svn-trunk-source + svn revert -R . ++ egrep -v '^X|^Performing status on external' ++ awk '{print $2}' ++ svn status --no-ignore + rm -rf target datanucleus.log ant/target shims/target shims/0.20/target shims/0.20S/target shims/0.23/target shims/aggregator/target shims/common/target shims/common-secure/target packaging/target hbase-handler/target testutils/target jdbc/target metastore/target itests/target itests/hcatalog-unit/target itests/test-serde/target itests/qtest/target itests/hive-unit/target itests/custom-serde/target itests/util/target hcatalog/target hcatalog/storage-handlers/hbase/target hcatalog/server-extensions/target hcatalog/core/target hcatalog/webhcat/svr/target hcatalog/webhcat/java-client/target hcatalog/hcatalog-pig-adapter/target hwi/target common/target common/src/gen service/target contrib/target serde/target beeline/target odbc/target cli/target ql/dependency-reduced-pom.xml ql/target + svn update Fetching external item into 'hcatalog/src/test/e2e/harness' External at revision 1557954. At revision 1557954. + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hive-ptest/working/scratch/build.patch + [[ -f /data/hive-ptest/working/scratch/build.patch ]] + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch The patch does not appear to apply with p0, p1, or p2 + exit 1 ' This message is automatically generated. ATTACHMENT ID: 12622722
        Hide
        Sergey Shelukhin added a comment -

        Hmm, I was able to merge and to apply patch w/o problems. Let's try again

        Show
        Sergey Shelukhin added a comment - Hmm, I was able to merge and to apply patch w/o problems. Let's try again
        Hide
        Hive QA added a comment -

        Overall: +1 all checks pass

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12622970/HIVE-5951.07.patch

        SUCCESS: +1 4924 tests passed

        Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/910/testReport
        Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/910/console

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        

        This message is automatically generated.

        ATTACHMENT ID: 12622970

        Show
        Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12622970/HIVE-5951.07.patch SUCCESS: +1 4924 tests passed Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/910/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/910/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated. ATTACHMENT ID: 12622970
        Hide
        Ashutosh Chauhan added a comment -

        Committed to trunk. Thanks, Sergey!

        Show
        Ashutosh Chauhan added a comment - Committed to trunk. Thanks, Sergey!

          People

          • Assignee:
            Sergey Shelukhin
            Reporter:
            Sergey Shelukhin
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development