Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-14841 Replication - Phase 2
  3. HIVE-17428

REPL LOAD of ALTER_PARTITION event doesn't create import tasks if the partition doesn't exist during analyze phase.

    Details

      Description

      If the incremental dump event sequence have ADD_PARTITION followed by ALTER_PARTITION doesn't create any task for ALTER_PARTITION event as the partition doesn't exist during analyze phase. Due to this REPL STATUS returns wrong last repl ID.
      Scenario:
      1. Create DB
      2. Create partitioned table.
      3. Bootstrap dump and load
      4. Insert into table to a dynamically created partition. - This insert generate ADD_PARTITION and ALTER_PARTITION events.
      5. Incremental dump and load.

      • Load will be successful.
      • But the last repl ID set was incorrect as ALTER_PARTITION event was never applied.
      1. HIVE-17428.01.patch
        13 kB
        Sankar Hariappan
      2. HIVE-17428.02.patch
        20 kB
        Sankar Hariappan
      3. HIVE-17428.03.patch
        22 kB
        Sankar Hariappan

        Issue Links

          Activity

          Hide
          sankarh Sankar Hariappan added a comment -

          Added 01.patch with changes to add alterPartition task when a partition is missing during analyse phase.

          Show
          sankarh Sankar Hariappan added a comment - Added 01.patch with changes to add alterPartition task when a partition is missing during analyse phase.
          Hide
          hiveqa Hive QA added a comment -

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12885096/HIVE-17428.01.patch

          SUCCESS: +1 due to 1 test(s) being added or modified.

          ERROR: -1 due to 9 failed/errored test(s), 10990 tests executed
          Failed tests:

          TestLocationQueries - did not produce a TEST-*.xml file (likely timed out) (batchId=218)
          TestReplicationScenarios - did not produce a TEST-*.xml file (likely timed out) (batchId=218)
          TestReplicationScenariosAcrossInstances - did not produce a TEST-*.xml file (likely timed out) (batchId=218)
          TestSemanticAnalyzerHookLoading - did not produce a TEST-*.xml file (likely timed out) (batchId=218)
          TestTxnCommandsBase - did not produce a TEST-*.xml file (likely timed out) (batchId=280)
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=61)
          org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning] (batchId=169)
          org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=234)
          org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] (batchId=234)
          

          Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6658/testReport
          Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6658/console
          Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6658/

          Messages:

          Executing org.apache.hive.ptest.execution.TestCheckPhase
          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 9 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12885096 - PreCommit-HIVE-Build

          Show
          hiveqa Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12885096/HIVE-17428.01.patch SUCCESS: +1 due to 1 test(s) being added or modified. ERROR: -1 due to 9 failed/errored test(s), 10990 tests executed Failed tests: TestLocationQueries - did not produce a TEST-*.xml file (likely timed out) (batchId=218) TestReplicationScenarios - did not produce a TEST-*.xml file (likely timed out) (batchId=218) TestReplicationScenariosAcrossInstances - did not produce a TEST-*.xml file (likely timed out) (batchId=218) TestSemanticAnalyzerHookLoading - did not produce a TEST-*.xml file (likely timed out) (batchId=218) TestTxnCommandsBase - did not produce a TEST-*.xml file (likely timed out) (batchId=280) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=61) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning] (batchId=169) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=234) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] (batchId=234) Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6658/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6658/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6658/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 9 tests failed This message is automatically generated. ATTACHMENT ID: 12885096 - PreCommit-HIVE-Build
          Hide
          sankarh Sankar Hariappan added a comment -

          Added 02.patch with fixes for test failures.

          Show
          sankarh Sankar Hariappan added a comment - Added 02.patch with fixes for test failures.
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user sankarh opened a pull request:

          https://github.com/apache/hive/pull/243

          HIVE-17428: REPL LOAD of ALTER_PARTITION event doesn't create import tasks if the partition doesn't exist during analyze phase.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/sankarh/hive HIVE-17428

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/hive/pull/243.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #243


          commit 6838a6f9777aec97dcd204d295b3ba4179986e6b
          Author: Sankar Hariappan <mailtosankarh@gmail.com>
          Date: 2017-09-05T17:25:57Z

          HIVE-17428: REPL LOAD of ALTER_PARTITION event doesn't create import tasks if the partition doesn't exist during analyze phase.


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user sankarh opened a pull request: https://github.com/apache/hive/pull/243 HIVE-17428 : REPL LOAD of ALTER_PARTITION event doesn't create import tasks if the partition doesn't exist during analyze phase. You can merge this pull request into a Git repository by running: $ git pull https://github.com/sankarh/hive HIVE-17428 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hive/pull/243.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #243 commit 6838a6f9777aec97dcd204d295b3ba4179986e6b Author: Sankar Hariappan <mailtosankarh@gmail.com> Date: 2017-09-05T17:25:57Z HIVE-17428 : REPL LOAD of ALTER_PARTITION event doesn't create import tasks if the partition doesn't exist during analyze phase.
          Hide
          hiveqa Hive QA added a comment -

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12885427/HIVE-17428.02.patch

          SUCCESS: +1 due to 1 test(s) being added or modified.

          ERROR: -1 due to 5 failed/errored test(s), 11026 tests executed
          Failed tests:

          TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=230)
          TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=230)
          TestTxnCommandsBase - did not produce a TEST-*.xml file (likely timed out) (batchId=280)
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=61)
          org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=234)
          

          Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6677/testReport
          Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6677/console
          Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6677/

          Messages:

          Executing org.apache.hive.ptest.execution.TestCheckPhase
          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 5 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12885427 - PreCommit-HIVE-Build

          Show
          hiveqa Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12885427/HIVE-17428.02.patch SUCCESS: +1 due to 1 test(s) being added or modified. ERROR: -1 due to 5 failed/errored test(s), 11026 tests executed Failed tests: TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=230) TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=230) TestTxnCommandsBase - did not produce a TEST-*.xml file (likely timed out) (batchId=280) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=61) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=234) Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6677/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6677/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6677/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 5 tests failed This message is automatically generated. ATTACHMENT ID: 12885427 - PreCommit-HIVE-Build
          Hide
          sankarh Sankar Hariappan added a comment -

          Test failures are irrelevant to the patch. Re-submitted the same patch to see if any test suit times out.

          Show
          sankarh Sankar Hariappan added a comment - Test failures are irrelevant to the patch. Re-submitted the same patch to see if any test suit times out.
          Hide
          hiveqa Hive QA added a comment -

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12885570/HIVE-17428.02.patch

          SUCCESS: +1 due to 1 test(s) being added or modified.

          ERROR: -1 due to 8 failed/errored test(s), 11027 tests executed
          Failed tests:

          TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=230)
          TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=230)
          TestTxnCommandsBase - did not produce a TEST-*.xml file (likely timed out) (batchId=280)
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2] (batchId=46)
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cte_mat_4] (batchId=6)
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=61)
          org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=234)
          org.apache.hive.jdbc.TestJdbcWithMiniHS2.testHttpRetryOnServerIdleTimeout (batchId=227)
          

          Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6692/testReport
          Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6692/console
          Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6692/

          Messages:

          Executing org.apache.hive.ptest.execution.TestCheckPhase
          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 8 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12885570 - PreCommit-HIVE-Build

          Show
          hiveqa Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12885570/HIVE-17428.02.patch SUCCESS: +1 due to 1 test(s) being added or modified. ERROR: -1 due to 8 failed/errored test(s), 11027 tests executed Failed tests: TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=230) TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=230) TestTxnCommandsBase - did not produce a TEST-*.xml file (likely timed out) (batchId=280) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2] (batchId=46) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cte_mat_4] (batchId=6) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=61) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=234) org.apache.hive.jdbc.TestJdbcWithMiniHS2.testHttpRetryOnServerIdleTimeout (batchId=227) Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6692/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6692/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6692/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 8 tests failed This message is automatically generated. ATTACHMENT ID: 12885570 - PreCommit-HIVE-Build
          Hide
          sankarh Sankar Hariappan added a comment -

          TestAccumuloCliDriver, TestDummy and TestTxnCommandsBase are passed locally. Also, they are irrelevant to this patch.

          Show
          sankarh Sankar Hariappan added a comment - TestAccumuloCliDriver, TestDummy and TestTxnCommandsBase are passed locally. Also, they are irrelevant to this patch.
          Hide
          sankarh Sankar Hariappan added a comment -

          Added 03.patch after rebasing with master.

          Show
          sankarh Sankar Hariappan added a comment - Added 03.patch after rebasing with master.
          Hide
          anishek anishek added a comment -
          Show
          anishek anishek added a comment - +1 cc Thejas M Nair / Daniel Dai
          Hide
          hiveqa Hive QA added a comment -

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12886884/HIVE-17428.03.patch

          SUCCESS: +1 due to 1 test(s) being added or modified.

          ERROR: -1 due to 9 failed/errored test(s), 11040 tests executed
          Failed tests:

          TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=230)
          TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=230)
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_view] (batchId=39)
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=61)
          org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1] (batchId=143)
          org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats] (batchId=156)
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[drop_table_failure2] (batchId=89)
          org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=215)
          org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01 (batchId=215)
          

          Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6812/testReport
          Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6812/console
          Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6812/

          Messages:

          Executing org.apache.hive.ptest.execution.TestCheckPhase
          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 9 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12886884 - PreCommit-HIVE-Build

          Show
          hiveqa Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12886884/HIVE-17428.03.patch SUCCESS: +1 due to 1 test(s) being added or modified. ERROR: -1 due to 9 failed/errored test(s), 11040 tests executed Failed tests: TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=230) TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=230) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_view] (batchId=39) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=61) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1] (batchId=143) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats] (batchId=156) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[drop_table_failure2] (batchId=89) org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=215) org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01 (batchId=215) Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6812/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6812/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6812/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 9 tests failed This message is automatically generated. ATTACHMENT ID: 12886884 - PreCommit-HIVE-Build
          Hide
          thejas Thejas M Nair added a comment -

          +1

          Show
          thejas Thejas M Nair added a comment - +1
          Hide
          thejas Thejas M Nair added a comment -

          Patch committed to master.
          Thanks for the patch Sankar Hariappan and for the review anishek!

          Show
          thejas Thejas M Nair added a comment - Patch committed to master. Thanks for the patch Sankar Hariappan and for the review anishek !
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user sankarh closed the pull request at:

          https://github.com/apache/hive/pull/243

          Show
          githubbot ASF GitHub Bot added a comment - Github user sankarh closed the pull request at: https://github.com/apache/hive/pull/243

            People

            • Assignee:
              sankarh Sankar Hariappan
              Reporter:
              sankarh Sankar Hariappan
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development