Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-6133

Support partial partition exchange

    Details

    • Type: Improvement
    • Status: Patch Available
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Query Processor
    • Labels:
      None

      Description

      Current alter exchange coerces source and destination table to have same partition columns. But source table has sub-set of partitions and provided partition spec supplements to be a complete partition spec, it need not to be that.

      For example, table into partition

      CREATE TABLE exchange_part_test1 (f1 string) PARTITIONED BY (ds STRING);
      CREATE TABLE exchange_part_test2 (f1 string) 
      ALTER TABLE exchange_part_test1 EXCHANGE PARTITION (ds='2013-04-05') WITH TABLE exchange_part_test2;
      

      or

      partial partitions into parent partition

      CREATE TABLE exchange_part_test1 (f1 string) PARTITIONED BY (ds STRING, hr STRING);
      CREATE TABLE exchange_part_test2 (f1 string) PARTITIONED BY (hr STRING)
      ALTER TABLE exchange_part_test1 EXCHANGE PARTITION (ds='2013-04-05') WITH TABLE exchange_part_test2;
      

      can be possible.

        Issue Links

          Activity

          Hide
          hiveqa Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12638030/HIVE-6133.1.patch.txt

          ERROR: -1 due to 1 failed/errored test(s), 5520 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_dyn_part
          

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2070/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2070/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 1 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12638030

          Show
          hiveqa Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12638030/HIVE-6133.1.patch.txt ERROR: -1 due to 1 failed/errored test(s), 5520 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_dyn_part Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2070/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2070/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed This message is automatically generated. ATTACHMENT ID: 12638030
          Hide
          ngangam Naveen Gangam added a comment -

          Navis Should we revisit this improvement? From this feature documentation page at https://cwiki.apache.org/confluence/display/Hive/Exchange+Partition,
          When the command is executed, the source table's partition folder in HDFS will be renamed to move it to the destination table's partition folder.

          When exchanging a partition into a already partitioned table, what will the order of the partition keys be for the destination table? What will happen to existing partitions on the source table that only had single partition key when it was created? Thanks

          Show
          ngangam Naveen Gangam added a comment - Navis Should we revisit this improvement? From this feature documentation page at https://cwiki.apache.org/confluence/display/Hive/Exchange+Partition , When the command is executed, the source table's partition folder in HDFS will be renamed to move it to the destination table's partition folder. When exchanging a partition into a already partitioned table, what will the order of the partition keys be for the destination table? What will happen to existing partitions on the source table that only had single partition key when it was created? Thanks
          Hide
          hiveqa Hive QA added a comment -

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12638030/HIVE-6133.1.patch.txt

          ERROR: -1 due to build exiting with an error

          Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5756/testReport
          Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5756/console
          Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5756/

          Messages:

          Executing org.apache.hive.ptest.execution.TestCheckPhase
          Executing org.apache.hive.ptest.execution.PrepPhase
          Tests exited with: NonZeroExitCodeException
          Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N'
          2017-06-23 22:24:21.227
          + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
          + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
          + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
          + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
          + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
          + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
          + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
          + export 'MAVEN_OPTS=-Xmx1g '
          + MAVEN_OPTS='-Xmx1g '
          + cd /data/hiveptest/working/
          + tee /data/hiveptest/logs/PreCommit-HIVE-Build-5756/source-prep.txt
          + [[ false == \t\r\u\e ]]
          + mkdir -p maven ivy
          + [[ git = \s\v\n ]]
          + [[ git = \g\i\t ]]
          + [[ -z master ]]
          + [[ -d apache-github-source-source ]]
          + [[ ! -d apache-github-source-source/.git ]]
          + [[ ! -d apache-github-source-source ]]
          + date '+%Y-%m-%d %T.%3N'
          2017-06-23 22:24:21.229
          + cd apache-github-source-source
          + git fetch origin
          + git reset --hard HEAD
          HEAD is now at bc510f6 HIVE-16751: Support different types for grouping columns in GroupBy Druid queries (Jesus Camacho Rodriguez, reviewed by Ashutosh Chauhan)
          + git clean -f -d
          + git checkout master
          Already on 'master'
          Your branch is up-to-date with 'origin/master'.
          + git reset --hard origin/master
          HEAD is now at bc510f6 HIVE-16751: Support different types for grouping columns in GroupBy Druid queries (Jesus Camacho Rodriguez, reviewed by Ashutosh Chauhan)
          + git merge --ff-only origin/master
          Already up-to-date.
          + date '+%Y-%m-%d %T.%3N'
          2017-06-23 22:24:22.426
          + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
          + patchFilePath=/data/hiveptest/working/scratch/build.patch
          + [[ -f /data/hiveptest/working/scratch/build.patch ]]
          + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
          + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch
          error: patch failed: metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:120
          error: metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java: patch does not apply
          error: patch failed: metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java:1431
          error: metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java: patch does not apply
          error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java:641
          error: ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java: patch does not apply
          The patch does not appear to apply with p0, p1, or p2
          + exit 1
          '
          

          This message is automatically generated.

          ATTACHMENT ID: 12638030 - PreCommit-HIVE-Build

          Show
          hiveqa Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12638030/HIVE-6133.1.patch.txt ERROR: -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5756/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5756/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5756/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N' 2017-06-23 22:24:21.227 + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]] + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'MAVEN_OPTS=-Xmx1g ' + MAVEN_OPTS='-Xmx1g ' + cd /data/hiveptest/working/ + tee /data/hiveptest/logs/PreCommit-HIVE-Build-5756/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + date '+%Y-%m-%d %T.%3N' 2017-06-23 22:24:21.229 + cd apache-github-source-source + git fetch origin + git reset --hard HEAD HEAD is now at bc510f6 HIVE-16751: Support different types for grouping columns in GroupBy Druid queries (Jesus Camacho Rodriguez, reviewed by Ashutosh Chauhan) + git clean -f -d + git checkout master Already on 'master' Your branch is up-to-date with 'origin/master'. + git reset --hard origin/master HEAD is now at bc510f6 HIVE-16751: Support different types for grouping columns in GroupBy Druid queries (Jesus Camacho Rodriguez, reviewed by Ashutosh Chauhan) + git merge --ff-only origin/master Already up-to-date. + date '+%Y-%m-%d %T.%3N' 2017-06-23 22:24:22.426 + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hiveptest/working/scratch/build.patch + [[ -f /data/hiveptest/working/scratch/build.patch ]] + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch error: patch failed: metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:120 error: metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java: patch does not apply error: patch failed: metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java:1431 error: metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java: patch does not apply error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java:641 error: ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java: patch does not apply The patch does not appear to apply with p0, p1, or p2 + exit 1 ' This message is automatically generated. ATTACHMENT ID: 12638030 - PreCommit-HIVE-Build

            People

            • Assignee:
              navis Navis
              Reporter:
              navis Navis
            • Votes:
              2 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:

                Development