Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-19249

Replication: WITH clause is not passing the configuration to Task correctly in all cases

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 3.1.0, 3.0.0
    • 3.0.0
    • repl
    • None
    • Reviewed

    Description

      When running repl load like following:

      REPL LOAD `repldb_kms207` FROM 'hdfs://url:8020/apps/hive/repl/f8b057a7-c3f2-43bd-8baa-f7408a9008fc' WITH ('hive.exec.parallel'='true','hive.distcp.privileged.doAs'='beacon','hive.metastore.uris'='thrift://metastore-url:9083','hive.metastore.warehouse.dir'='s3a://s3-warehouse','hive.warehouse.subdir.inherit.perms'='false','hive.repl.replica.functions.root.dir'='s3a://s3-warehouse','fs.s3a.bucket.ss-datasets.endpoint'='s3-bucket-endpoint','fs.s3a.impl.disable.cache'='true','fs.s3a.server-side-encryption-algorithm'='SSE-KMS','fs.s3a.server-side-encryption.key'='encr-key','distcp.options.pp'='','distcp.options.pg'='','distcp.options.pu'='');
      

      the task that get created need to use the configs that are passed in the USING clause. However, in some cases the wrong config object gets used.

      Attachments

        1. HIVE-19249.1.patch
          5 kB
          Vaibhav Gumashta
        2. HIVE-19249.2.patch
          2 kB
          Vaibhav Gumashta

        Activity

          daijy Daniel Dai added a comment -

          Shall we restore the session conf after replication command?

          daijy Daniel Dai added a comment - Shall we restore the session conf after replication command?
          thejas Thejas Nair added a comment -

          +1 pending tests.

          I noticed that in master branch because of changes in HIVE-18238, every query already has a new HiveConf object. So we no longer need to create a new copy in ReplicationSemanticAnalyzer. This could be part of a follow up change (its not introduced by this patch).

          thejas Thejas Nair added a comment - +1 pending tests. I noticed that in master branch because of changes in HIVE-18238 , every query already has a new HiveConf object. So we no longer need to create a new copy in ReplicationSemanticAnalyzer. This could be part of a follow up change (its not introduced by this patch).
          hiveqa Hive QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
                Prechecks
          0 findbugs 0m 1s Findbugs executables are not available.
          +1 @author 0m 0s The patch does not contain any @author tags.
                master Compile Tests
          +1 mvninstall 8m 42s master passed
          +1 compile 1m 15s master passed
          +1 checkstyle 0m 44s master passed
          +1 javadoc 1m 2s master passed
                Patch Compile Tests
          +1 mvninstall 1m 43s the patch passed
          +1 compile 1m 12s the patch passed
          +1 javac 1m 12s the patch passed
          +1 checkstyle 0m 45s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 javadoc 1m 4s the patch passed
                Other Tests
          +1 asflicense 0m 15s The patch does not generate ASF License warnings.
          17m 4s



          Subsystem Report/Notes
          Optional Tests asflicense javac javadoc findbugs checkstyle compile
          uname Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux
          Build tool maven
          Personality /data/hiveptest/working/yetus_PreCommit-HIVE-Build-10372/dev-support/hive-personality.sh
          git revision master / 9cfaf6b
          Default Java 1.8.0_111
          modules C: ql U: ql
          Console output http://104.198.109.242/logs//PreCommit-HIVE-Build-10372/yetus.txt
          Powered by Apache Yetus http://yetus.apache.org

          This message was automatically generated.

          hiveqa Hive QA added a comment - +1 overall Vote Subsystem Runtime Comment       Prechecks 0 findbugs 0m 1s Findbugs executables are not available. +1 @author 0m 0s The patch does not contain any @author tags.       master Compile Tests +1 mvninstall 8m 42s master passed +1 compile 1m 15s master passed +1 checkstyle 0m 44s master passed +1 javadoc 1m 2s master passed       Patch Compile Tests +1 mvninstall 1m 43s the patch passed +1 compile 1m 12s the patch passed +1 javac 1m 12s the patch passed +1 checkstyle 0m 45s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 javadoc 1m 4s the patch passed       Other Tests +1 asflicense 0m 15s The patch does not generate ASF License warnings. 17m 4s Subsystem Report/Notes Optional Tests asflicense javac javadoc findbugs checkstyle compile uname Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux Build tool maven Personality /data/hiveptest/working/yetus_PreCommit-HIVE-Build-10372/dev-support/hive-personality.sh git revision master / 9cfaf6b Default Java 1.8.0_111 modules C: ql U: ql Console output http://104.198.109.242/logs//PreCommit-HIVE-Build-10372/yetus.txt Powered by Apache Yetus http://yetus.apache.org This message was automatically generated.
          hiveqa Hive QA added a comment -

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12919898/HIVE-19249.2.patch

          ERROR: -1 due to no test(s) being added or modified.

          ERROR: -1 due to 22 failed/errored test(s), 14280 tests executed
          Failed tests:

          TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=93)
          	[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
          TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed out) (batchId=217)
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_smb] (batchId=92)
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_0] (batchId=17)
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[results_cache_invalidation2] (batchId=39)
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[tez_join_hash] (batchId=54)
          org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_invalidation2] (batchId=163)
          org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] (batchId=163)
          org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_1] (batchId=171)
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5] (batchId=105)
          org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[cluster_tasklog_retrieval] (batchId=98)
          org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace] (batchId=98)
          org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace_turnoff] (batchId=98)
          org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[minimr_broken_pipe] (batchId=98)
          org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut (batchId=225)
          org.apache.hadoop.hive.ql.TestAcidOnTez.testAcidInsertWithRemoveUnion (batchId=228)
          org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=228)
          org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01 (batchId=228)
          org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 (batchId=232)
          org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgress (batchId=235)
          org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgressParallel (batchId=235)
          org.apache.hive.minikdc.TestJdbcWithMiniKdcCookie.testCookieNegative (batchId=254)
          

          Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/10372/testReport
          Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/10372/console
          Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-10372/

          Messages:

          Executing org.apache.hive.ptest.execution.TestCheckPhase
          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.YetusPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 22 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12919898 - PreCommit-HIVE-Build

          hiveqa Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12919898/HIVE-19249.2.patch ERROR: -1 due to no test(s) being added or modified. ERROR: -1 due to 22 failed/errored test(s), 14280 tests executed Failed tests: TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=93) [infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q] TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed out) (batchId=217) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_smb] (batchId=92) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_0] (batchId=17) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[results_cache_invalidation2] (batchId=39) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[tez_join_hash] (batchId=54) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_invalidation2] (batchId=163) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] (batchId=163) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_1] (batchId=171) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5] (batchId=105) org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[cluster_tasklog_retrieval] (batchId=98) org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace] (batchId=98) org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace_turnoff] (batchId=98) org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[minimr_broken_pipe] (batchId=98) org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut (batchId=225) org.apache.hadoop.hive.ql.TestAcidOnTez.testAcidInsertWithRemoveUnion (batchId=228) org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=228) org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01 (batchId=228) org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 (batchId=232) org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgress (batchId=235) org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgressParallel (batchId=235) org.apache.hive.minikdc.TestJdbcWithMiniKdcCookie.testCookieNegative (batchId=254) Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/10372/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/10372/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-10372/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 22 tests failed This message is automatically generated. ATTACHMENT ID: 12919898 - PreCommit-HIVE-Build

          Committed to branch-3 and master. Thanks thejas, daijy

          vgumashta Vaibhav Gumashta added a comment - Committed to branch-3 and master. Thanks thejas , daijy

          This jira is resolved and released with Hive 3.0 If you find an issue with it, please create a new jira.

          ashutoshc Ashutosh Chauhan added a comment - This jira is resolved and released with Hive 3.0 If you find an issue with it, please create a new jira.

          People

            vgumashta Vaibhav Gumashta
            vgumashta Vaibhav Gumashta
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: