Hive
  1. Hive
  2. HIVE-2278

Support archiving for multiple partitions if the table is partitioned by multiple columns

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Hide
      Multi-archives support for hive, complete implementation.
      https://reviews.apache.org/r/1259/
      Show
      Multi-archives support for hive, complete implementation. https://reviews.apache.org/r/1259/

      Description

      If a table is partitioned by ds,hr
      it should be possible to archive all the files in ds to reduce the number of files

      1. HIVE-2278.12.patch
        163 kB
        He Yongqiang
      2. HIVE-2278.11.patch
        163 kB
        He Yongqiang
      3. HIVE-2278.10.patch
        167 kB
        He Yongqiang
      4. HIVE-2278.9.patch
        167 kB
        He Yongqiang
      5. HIVE-2278.8.patch
        159 kB
        He Yongqiang
      6. HIVE-2278.6.patch
        159 kB
        He Yongqiang
      7. archive_corrupt.rc
        1 kB
        Marcin Kurczych
      8. HIVE-2278.7.patch
        165 kB
        Marcin Kurczych
      9. HIVE-2278.6.patch
        165 kB
        Marcin Kurczych
      10. HIVE-2278.5.patch
        164 kB
        Marcin Kurczych
      11. HIVE-2278.5.patch
        164 kB
        Marcin Kurczych
      12. HIVE-2278.4.patch
        162 kB
        Marcin Kurczych
      13. HIVE-2278.3.patch
        165 kB
        Marcin Kurczych
      14. HIVE-2278.2.patch
        164 kB
        Marcin Kurczych
      15. hive.2278.1.patch
        88 kB
        Marcin Kurczych

        Activity

        Hide
        Marcin Kurczych added a comment -

        First version of a patch, not really ready yet

        Show
        Marcin Kurczych added a comment - First version of a patch, not really ready yet
        Hide
        Marcin Kurczych added a comment -

        Curent version of patch

        Show
        Marcin Kurczych added a comment - Curent version of patch
        Hide
        Marcin Kurczych added a comment -

        Ready version of patch

        Show
        Marcin Kurczych added a comment - Ready version of patch
        Hide
        Carl Steinbach added a comment -

        I see failures in the following tests. Please take a look:

        org.apache.hadoop.hive.metastore.TestMetaStoreEventListener.testListener
        org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testSynchronized
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_archive_corrupt
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_archive_multi
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_2
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_view_partitioned
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_drop_multi_partitions
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_escape1
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_inputddl6

        Show
        Carl Steinbach added a comment - I see failures in the following tests. Please take a look: org.apache.hadoop.hive.metastore.TestMetaStoreEventListener.testListener org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testSynchronized org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_archive_corrupt org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_archive_multi org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_view_partitioned org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_drop_multi_partitions org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_escape1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_inputddl6
        Hide
        Marcin Kurczych added a comment -

        -> data/files/archive_corrupt.rc

        Show
        Marcin Kurczych added a comment - -> data/files/archive_corrupt.rc
        Hide
        Marcin Kurczych added a comment -

        Fixed following bugs (accidental line deletion in patch):
        org.apache.hadoop.hive.metastore.TestMetaStoreEventListener.testListener
        org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testSynchronized
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_2
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_view_partitioned
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_drop_multi_partitions
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_escape1
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_inputddl6

        Following ones depend on https://issues.apache.org/jira/browse/HIVE-2400 :
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_archive_corrupt
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_archive_multi

        Show
        Marcin Kurczych added a comment - Fixed following bugs (accidental line deletion in patch): org.apache.hadoop.hive.metastore.TestMetaStoreEventListener.testListener org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testSynchronized org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_view_partitioned org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_drop_multi_partitions org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_escape1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_inputddl6 Following ones depend on https://issues.apache.org/jira/browse/HIVE-2400 : org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_archive_corrupt org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_archive_multi
        Hide
        He Yongqiang added a comment -

        refresh

        Show
        He Yongqiang added a comment - refresh
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/2595/
        -----------------------------------------------------------

        Review request for hive and namit jain.

        Summary
        -------

        Support archiving for multiple partitions if the table is partitioned by multiple columns

        This addresses bug HIVE-2278.
        https://issues.apache.org/jira/browse/HIVE-2278

        Diffs


        trunk/data/files/archive_corrupt.rc UNKNOWN
        trunk/metastore/if/hive_metastore.thrift 1188906
        trunk/metastore/src/gen/thrift/gen-cpp/hive_metastore_constants.h 1188906
        trunk/metastore/src/gen/thrift/gen-cpp/hive_metastore_constants.cpp 1188906
        trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Constants.java 1188906
        trunk/metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_constants.php 1188906
        trunk/metastore/src/gen/thrift/gen-py/hive_metastore/constants.py 1188906
        trunk/metastore/src/gen/thrift/gen-rb/hive_metastore_constants.rb 1188906
        trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1188906
        trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ArchiveUtils.java PRE-CREATION
        trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1188906
        trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1188906
        trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java 1188906
        trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/DummyPartition.java 1188906
        trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 1188906
        trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 1188906
        trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1188906
        trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1188906
        trunk/ql/src/test/queries/clientnegative/archive_insert1.q PRE-CREATION
        trunk/ql/src/test/queries/clientnegative/archive_insert2.q PRE-CREATION
        trunk/ql/src/test/queries/clientnegative/archive_insert3.q PRE-CREATION
        trunk/ql/src/test/queries/clientnegative/archive_insert4.q PRE-CREATION
        trunk/ql/src/test/queries/clientnegative/archive_multi1.q PRE-CREATION
        trunk/ql/src/test/queries/clientnegative/archive_multi2.q PRE-CREATION
        trunk/ql/src/test/queries/clientnegative/archive_multi3.q PRE-CREATION
        trunk/ql/src/test/queries/clientnegative/archive_multi4.q PRE-CREATION
        trunk/ql/src/test/queries/clientnegative/archive_multi5.q PRE-CREATION
        trunk/ql/src/test/queries/clientnegative/archive_multi6.q PRE-CREATION
        trunk/ql/src/test/queries/clientnegative/archive_multi7.q PRE-CREATION
        trunk/ql/src/test/queries/clientnegative/archive_partspec1.q PRE-CREATION
        trunk/ql/src/test/queries/clientnegative/archive_partspec2.q PRE-CREATION
        trunk/ql/src/test/queries/clientnegative/archive_partspec3.q PRE-CREATION
        trunk/ql/src/test/queries/clientpositive/archive_corrupt.q PRE-CREATION
        trunk/ql/src/test/queries/clientpositive/archive_multi.q PRE-CREATION
        trunk/ql/src/test/results/clientnegative/archive1.q.out 1188906
        trunk/ql/src/test/results/clientnegative/archive2.q.out 1188906
        trunk/ql/src/test/results/clientnegative/archive_insert1.q.out PRE-CREATION
        trunk/ql/src/test/results/clientnegative/archive_insert2.q.out PRE-CREATION
        trunk/ql/src/test/results/clientnegative/archive_insert3.q.out PRE-CREATION
        trunk/ql/src/test/results/clientnegative/archive_insert4.q.out PRE-CREATION
        trunk/ql/src/test/results/clientnegative/archive_multi1.q.out PRE-CREATION
        trunk/ql/src/test/results/clientnegative/archive_multi2.q.out PRE-CREATION
        trunk/ql/src/test/results/clientnegative/archive_multi3.q.out PRE-CREATION
        trunk/ql/src/test/results/clientnegative/archive_multi4.q.out PRE-CREATION
        trunk/ql/src/test/results/clientnegative/archive_multi5.q.out PRE-CREATION
        trunk/ql/src/test/results/clientnegative/archive_multi6.q.out PRE-CREATION
        trunk/ql/src/test/results/clientnegative/archive_multi7.q.out PRE-CREATION
        trunk/ql/src/test/results/clientnegative/archive_partspec1.q.out PRE-CREATION
        trunk/ql/src/test/results/clientnegative/archive_partspec2.q.out PRE-CREATION
        trunk/ql/src/test/results/clientnegative/archive_partspec3.q.out PRE-CREATION
        trunk/ql/src/test/results/clientpositive/archive_corrupt.q.out PRE-CREATION
        trunk/ql/src/test/results/clientpositive/archive_multi.q.out PRE-CREATION

        Diff: https://reviews.apache.org/r/2595/diff

        Testing
        -------

        Support archiving for multiple partitions if the table is partitioned by multiple columns

        Thanks,

        Yongqiang

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2595/ ----------------------------------------------------------- Review request for hive and namit jain. Summary ------- Support archiving for multiple partitions if the table is partitioned by multiple columns This addresses bug HIVE-2278 . https://issues.apache.org/jira/browse/HIVE-2278 Diffs trunk/data/files/archive_corrupt.rc UNKNOWN trunk/metastore/if/hive_metastore.thrift 1188906 trunk/metastore/src/gen/thrift/gen-cpp/hive_metastore_constants.h 1188906 trunk/metastore/src/gen/thrift/gen-cpp/hive_metastore_constants.cpp 1188906 trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Constants.java 1188906 trunk/metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_constants.php 1188906 trunk/metastore/src/gen/thrift/gen-py/hive_metastore/constants.py 1188906 trunk/metastore/src/gen/thrift/gen-rb/hive_metastore_constants.rb 1188906 trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1188906 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ArchiveUtils.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1188906 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1188906 trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java 1188906 trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/DummyPartition.java 1188906 trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 1188906 trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 1188906 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1188906 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1188906 trunk/ql/src/test/queries/clientnegative/archive_insert1.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_insert2.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_insert3.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_insert4.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_multi1.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_multi2.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_multi3.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_multi4.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_multi5.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_multi6.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_multi7.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_partspec1.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_partspec2.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_partspec3.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/archive_corrupt.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/archive_multi.q PRE-CREATION trunk/ql/src/test/results/clientnegative/archive1.q.out 1188906 trunk/ql/src/test/results/clientnegative/archive2.q.out 1188906 trunk/ql/src/test/results/clientnegative/archive_insert1.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_insert2.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_insert3.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_insert4.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_multi1.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_multi2.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_multi3.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_multi4.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_multi5.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_multi6.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_multi7.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_partspec1.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_partspec2.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_partspec3.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/archive_corrupt.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/archive_multi.q.out PRE-CREATION Diff: https://reviews.apache.org/r/2595/diff Testing ------- Support archiving for multiple partitions if the table is partitioned by multiple columns Thanks, Yongqiang
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/2595/#review2915
        -----------------------------------------------------------

        what happens to external tables ?

        If TabAExt/ds=1 points to /WH/TabA/ds=1, and we archive TabA (ds=1),
        isn't TabAExt also getting ruined ? Or, is it the burden of the owner of the
        external table to fix the location of TabA (ds=1)

        trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ArchiveUtils.java
        <https://reviews.apache.org/r/2595/#comment6517>

        wrong comment

        trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ArchiveUtils.java
        <https://reviews.apache.org/r/2595/#comment6518>

        what happens if the order of columns in partSpec is wrong

        trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java
        <https://reviews.apache.org/r/2595/#comment6516>

        This logic looks wrong - dont you need to check the
        order of columns etc.

        • namit

        On 2011-10-28 18:28:45, Yongqiang He wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/2595/

        -----------------------------------------------------------

        (Updated 2011-10-28 18:28:45)

        Review request for hive and namit jain.

        Summary

        -------

        Support archiving for multiple partitions if the table is partitioned by multiple columns

        This addresses bug HIVE-2278.

        https://issues.apache.org/jira/browse/HIVE-2278

        Diffs

        -----

        trunk/data/files/archive_corrupt.rc UNKNOWN

        trunk/metastore/if/hive_metastore.thrift 1188906

        trunk/metastore/src/gen/thrift/gen-cpp/hive_metastore_constants.h 1188906

        trunk/metastore/src/gen/thrift/gen-cpp/hive_metastore_constants.cpp 1188906

        trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Constants.java 1188906

        trunk/metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_constants.php 1188906

        trunk/metastore/src/gen/thrift/gen-py/hive_metastore/constants.py 1188906

        trunk/metastore/src/gen/thrift/gen-rb/hive_metastore_constants.rb 1188906

        trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1188906

        trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ArchiveUtils.java PRE-CREATION

        trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1188906

        trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1188906

        trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java 1188906

        trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/DummyPartition.java 1188906

        trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 1188906

        trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 1188906

        trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1188906

        trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1188906

        trunk/ql/src/test/queries/clientnegative/archive_insert1.q PRE-CREATION

        trunk/ql/src/test/queries/clientnegative/archive_insert2.q PRE-CREATION

        trunk/ql/src/test/queries/clientnegative/archive_insert3.q PRE-CREATION

        trunk/ql/src/test/queries/clientnegative/archive_insert4.q PRE-CREATION

        trunk/ql/src/test/queries/clientnegative/archive_multi1.q PRE-CREATION

        trunk/ql/src/test/queries/clientnegative/archive_multi2.q PRE-CREATION

        trunk/ql/src/test/queries/clientnegative/archive_multi3.q PRE-CREATION

        trunk/ql/src/test/queries/clientnegative/archive_multi4.q PRE-CREATION

        trunk/ql/src/test/queries/clientnegative/archive_multi5.q PRE-CREATION

        trunk/ql/src/test/queries/clientnegative/archive_multi6.q PRE-CREATION

        trunk/ql/src/test/queries/clientnegative/archive_multi7.q PRE-CREATION

        trunk/ql/src/test/queries/clientnegative/archive_partspec1.q PRE-CREATION

        trunk/ql/src/test/queries/clientnegative/archive_partspec2.q PRE-CREATION

        trunk/ql/src/test/queries/clientnegative/archive_partspec3.q PRE-CREATION

        trunk/ql/src/test/queries/clientpositive/archive_corrupt.q PRE-CREATION

        trunk/ql/src/test/queries/clientpositive/archive_multi.q PRE-CREATION

        trunk/ql/src/test/results/clientnegative/archive1.q.out 1188906

        trunk/ql/src/test/results/clientnegative/archive2.q.out 1188906

        trunk/ql/src/test/results/clientnegative/archive_insert1.q.out PRE-CREATION

        trunk/ql/src/test/results/clientnegative/archive_insert2.q.out PRE-CREATION

        trunk/ql/src/test/results/clientnegative/archive_insert3.q.out PRE-CREATION

        trunk/ql/src/test/results/clientnegative/archive_insert4.q.out PRE-CREATION

        trunk/ql/src/test/results/clientnegative/archive_multi1.q.out PRE-CREATION

        trunk/ql/src/test/results/clientnegative/archive_multi2.q.out PRE-CREATION

        trunk/ql/src/test/results/clientnegative/archive_multi3.q.out PRE-CREATION

        trunk/ql/src/test/results/clientnegative/archive_multi4.q.out PRE-CREATION

        trunk/ql/src/test/results/clientnegative/archive_multi5.q.out PRE-CREATION

        trunk/ql/src/test/results/clientnegative/archive_multi6.q.out PRE-CREATION

        trunk/ql/src/test/results/clientnegative/archive_multi7.q.out PRE-CREATION

        trunk/ql/src/test/results/clientnegative/archive_partspec1.q.out PRE-CREATION

        trunk/ql/src/test/results/clientnegative/archive_partspec2.q.out PRE-CREATION

        trunk/ql/src/test/results/clientnegative/archive_partspec3.q.out PRE-CREATION

        trunk/ql/src/test/results/clientpositive/archive_corrupt.q.out PRE-CREATION

        trunk/ql/src/test/results/clientpositive/archive_multi.q.out PRE-CREATION

        Diff: https://reviews.apache.org/r/2595/diff

        Testing

        -------

        Support archiving for multiple partitions if the table is partitioned by multiple columns

        Thanks,

        Yongqiang

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2595/#review2915 ----------------------------------------------------------- what happens to external tables ? If TabAExt/ds=1 points to /WH/TabA/ds=1, and we archive TabA (ds=1), isn't TabAExt also getting ruined ? Or, is it the burden of the owner of the external table to fix the location of TabA (ds=1) trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ArchiveUtils.java < https://reviews.apache.org/r/2595/#comment6517 > wrong comment trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ArchiveUtils.java < https://reviews.apache.org/r/2595/#comment6518 > what happens if the order of columns in partSpec is wrong trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java < https://reviews.apache.org/r/2595/#comment6516 > This logic looks wrong - dont you need to check the order of columns etc. namit On 2011-10-28 18:28:45, Yongqiang He wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2595/ ----------------------------------------------------------- (Updated 2011-10-28 18:28:45) Review request for hive and namit jain. Summary ------- Support archiving for multiple partitions if the table is partitioned by multiple columns This addresses bug HIVE-2278 . https://issues.apache.org/jira/browse/HIVE-2278 Diffs ----- trunk/data/files/archive_corrupt.rc UNKNOWN trunk/metastore/if/hive_metastore.thrift 1188906 trunk/metastore/src/gen/thrift/gen-cpp/hive_metastore_constants.h 1188906 trunk/metastore/src/gen/thrift/gen-cpp/hive_metastore_constants.cpp 1188906 trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Constants.java 1188906 trunk/metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_constants.php 1188906 trunk/metastore/src/gen/thrift/gen-py/hive_metastore/constants.py 1188906 trunk/metastore/src/gen/thrift/gen-rb/hive_metastore_constants.rb 1188906 trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1188906 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ArchiveUtils.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1188906 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1188906 trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java 1188906 trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/DummyPartition.java 1188906 trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 1188906 trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 1188906 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1188906 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1188906 trunk/ql/src/test/queries/clientnegative/archive_insert1.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_insert2.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_insert3.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_insert4.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_multi1.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_multi2.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_multi3.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_multi4.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_multi5.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_multi6.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_multi7.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_partspec1.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_partspec2.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_partspec3.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/archive_corrupt.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/archive_multi.q PRE-CREATION trunk/ql/src/test/results/clientnegative/archive1.q.out 1188906 trunk/ql/src/test/results/clientnegative/archive2.q.out 1188906 trunk/ql/src/test/results/clientnegative/archive_insert1.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_insert2.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_insert3.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_insert4.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_multi1.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_multi2.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_multi3.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_multi4.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_multi5.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_multi6.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_multi7.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_partspec1.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_partspec2.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_partspec3.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/archive_corrupt.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/archive_multi.q.out PRE-CREATION Diff: https://reviews.apache.org/r/2595/diff Testing ------- Support archiving for multiple partitions if the table is partitioned by multiple columns Thanks, Yongqiang
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/2595/#review2918
        -----------------------------------------------------------

        trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ArchiveUtils.java
        <https://reviews.apache.org/r/2595/#comment6526>

        In the example above, (hr='1', ds='2011-01-01') will work. It should not work - it would be simpler if you check the order also

        trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ArchiveUtils.java
        <https://reviews.apache.org/r/2595/#comment6527>

        add some comments

        trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ArchiveUtils.java
        <https://reviews.apache.org/r/2595/#comment6528>

        1 for loop instead of 2 for loops

        trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ArchiveUtils.java
        <https://reviews.apache.org/r/2595/#comment6529>

        I think there is a bug here - not sure

        Why have a for loop - you are returning inside if anyway the first time ?

        • namit

        On 2011-10-28 18:28:45, Yongqiang He wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/2595/

        -----------------------------------------------------------

        (Updated 2011-10-28 18:28:45)

        Review request for hive and namit jain.

        Summary

        -------

        Support archiving for multiple partitions if the table is partitioned by multiple columns

        This addresses bug HIVE-2278.

        https://issues.apache.org/jira/browse/HIVE-2278

        Diffs

        -----

        trunk/data/files/archive_corrupt.rc UNKNOWN

        trunk/metastore/if/hive_metastore.thrift 1188906

        trunk/metastore/src/gen/thrift/gen-cpp/hive_metastore_constants.h 1188906

        trunk/metastore/src/gen/thrift/gen-cpp/hive_metastore_constants.cpp 1188906

        trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Constants.java 1188906

        trunk/metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_constants.php 1188906

        trunk/metastore/src/gen/thrift/gen-py/hive_metastore/constants.py 1188906

        trunk/metastore/src/gen/thrift/gen-rb/hive_metastore_constants.rb 1188906

        trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1188906

        trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ArchiveUtils.java PRE-CREATION

        trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1188906

        trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1188906

        trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java 1188906

        trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/DummyPartition.java 1188906

        trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 1188906

        trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 1188906

        trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1188906

        trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1188906

        trunk/ql/src/test/queries/clientnegative/archive_insert1.q PRE-CREATION

        trunk/ql/src/test/queries/clientnegative/archive_insert2.q PRE-CREATION

        trunk/ql/src/test/queries/clientnegative/archive_insert3.q PRE-CREATION

        trunk/ql/src/test/queries/clientnegative/archive_insert4.q PRE-CREATION

        trunk/ql/src/test/queries/clientnegative/archive_multi1.q PRE-CREATION

        trunk/ql/src/test/queries/clientnegative/archive_multi2.q PRE-CREATION

        trunk/ql/src/test/queries/clientnegative/archive_multi3.q PRE-CREATION

        trunk/ql/src/test/queries/clientnegative/archive_multi4.q PRE-CREATION

        trunk/ql/src/test/queries/clientnegative/archive_multi5.q PRE-CREATION

        trunk/ql/src/test/queries/clientnegative/archive_multi6.q PRE-CREATION

        trunk/ql/src/test/queries/clientnegative/archive_multi7.q PRE-CREATION

        trunk/ql/src/test/queries/clientnegative/archive_partspec1.q PRE-CREATION

        trunk/ql/src/test/queries/clientnegative/archive_partspec2.q PRE-CREATION

        trunk/ql/src/test/queries/clientnegative/archive_partspec3.q PRE-CREATION

        trunk/ql/src/test/queries/clientpositive/archive_corrupt.q PRE-CREATION

        trunk/ql/src/test/queries/clientpositive/archive_multi.q PRE-CREATION

        trunk/ql/src/test/results/clientnegative/archive1.q.out 1188906

        trunk/ql/src/test/results/clientnegative/archive2.q.out 1188906

        trunk/ql/src/test/results/clientnegative/archive_insert1.q.out PRE-CREATION

        trunk/ql/src/test/results/clientnegative/archive_insert2.q.out PRE-CREATION

        trunk/ql/src/test/results/clientnegative/archive_insert3.q.out PRE-CREATION

        trunk/ql/src/test/results/clientnegative/archive_insert4.q.out PRE-CREATION

        trunk/ql/src/test/results/clientnegative/archive_multi1.q.out PRE-CREATION

        trunk/ql/src/test/results/clientnegative/archive_multi2.q.out PRE-CREATION

        trunk/ql/src/test/results/clientnegative/archive_multi3.q.out PRE-CREATION

        trunk/ql/src/test/results/clientnegative/archive_multi4.q.out PRE-CREATION

        trunk/ql/src/test/results/clientnegative/archive_multi5.q.out PRE-CREATION

        trunk/ql/src/test/results/clientnegative/archive_multi6.q.out PRE-CREATION

        trunk/ql/src/test/results/clientnegative/archive_multi7.q.out PRE-CREATION

        trunk/ql/src/test/results/clientnegative/archive_partspec1.q.out PRE-CREATION

        trunk/ql/src/test/results/clientnegative/archive_partspec2.q.out PRE-CREATION

        trunk/ql/src/test/results/clientnegative/archive_partspec3.q.out PRE-CREATION

        trunk/ql/src/test/results/clientpositive/archive_corrupt.q.out PRE-CREATION

        trunk/ql/src/test/results/clientpositive/archive_multi.q.out PRE-CREATION

        Diff: https://reviews.apache.org/r/2595/diff

        Testing

        -------

        Support archiving for multiple partitions if the table is partitioned by multiple columns

        Thanks,

        Yongqiang

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2595/#review2918 ----------------------------------------------------------- trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ArchiveUtils.java < https://reviews.apache.org/r/2595/#comment6526 > In the example above, (hr='1', ds='2011-01-01') will work. It should not work - it would be simpler if you check the order also trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ArchiveUtils.java < https://reviews.apache.org/r/2595/#comment6527 > add some comments trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ArchiveUtils.java < https://reviews.apache.org/r/2595/#comment6528 > 1 for loop instead of 2 for loops trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ArchiveUtils.java < https://reviews.apache.org/r/2595/#comment6529 > I think there is a bug here - not sure Why have a for loop - you are returning inside if anyway the first time ? namit On 2011-10-28 18:28:45, Yongqiang He wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2595/ ----------------------------------------------------------- (Updated 2011-10-28 18:28:45) Review request for hive and namit jain. Summary ------- Support archiving for multiple partitions if the table is partitioned by multiple columns This addresses bug HIVE-2278 . https://issues.apache.org/jira/browse/HIVE-2278 Diffs ----- trunk/data/files/archive_corrupt.rc UNKNOWN trunk/metastore/if/hive_metastore.thrift 1188906 trunk/metastore/src/gen/thrift/gen-cpp/hive_metastore_constants.h 1188906 trunk/metastore/src/gen/thrift/gen-cpp/hive_metastore_constants.cpp 1188906 trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Constants.java 1188906 trunk/metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_constants.php 1188906 trunk/metastore/src/gen/thrift/gen-py/hive_metastore/constants.py 1188906 trunk/metastore/src/gen/thrift/gen-rb/hive_metastore_constants.rb 1188906 trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1188906 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ArchiveUtils.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1188906 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1188906 trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java 1188906 trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/DummyPartition.java 1188906 trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 1188906 trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 1188906 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1188906 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1188906 trunk/ql/src/test/queries/clientnegative/archive_insert1.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_insert2.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_insert3.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_insert4.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_multi1.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_multi2.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_multi3.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_multi4.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_multi5.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_multi6.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_multi7.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_partspec1.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_partspec2.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_partspec3.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/archive_corrupt.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/archive_multi.q PRE-CREATION trunk/ql/src/test/results/clientnegative/archive1.q.out 1188906 trunk/ql/src/test/results/clientnegative/archive2.q.out 1188906 trunk/ql/src/test/results/clientnegative/archive_insert1.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_insert2.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_insert3.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_insert4.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_multi1.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_multi2.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_multi3.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_multi4.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_multi5.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_multi6.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_multi7.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_partspec1.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_partspec2.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_partspec3.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/archive_corrupt.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/archive_multi.q.out PRE-CREATION Diff: https://reviews.apache.org/r/2595/diff Testing ------- Support archiving for multiple partitions if the table is partitioned by multiple columns Thanks, Yongqiang
        Hide
        Namit Jain added a comment -

        commented on the review board - looks good otherwise

        Show
        Namit Jain added a comment - commented on the review board - looks good otherwise
        Hide
        He Yongqiang added a comment -

        attached a new patch to address comments

        Show
        He Yongqiang added a comment - attached a new patch to address comments
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/2595/
        -----------------------------------------------------------

        (Updated 2011-10-30 07:34:36.866186)

        Review request for hive and namit jain.

        Changes
        -------

        Address review comments except the last one:
        "Why have a for loop - you are returning inside if anyway the first time ?"

        I think it is safe to return even for the first time. For any partition returned, it should either be archived or not. If the archive is based on its parent partition col, the partition should also be marked as archived, otherwise, it should not be there.

        Summary
        -------

        Support archiving for multiple partitions if the table is partitioned by multiple columns

        This addresses bug HIVE-2278.
        https://issues.apache.org/jira/browse/HIVE-2278

        Diffs (updated)


        trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1190490
        trunk/metastore/src/gen/thrift/gen-rb/hive_metastore_constants.rb 1190490
        trunk/metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_constants.php 1190490
        trunk/metastore/src/gen/thrift/gen-py/hive_metastore/constants.py 1190490
        trunk/data/files/archive_corrupt.rc UNKNOWN
        trunk/metastore/if/hive_metastore.thrift 1190490
        trunk/metastore/src/gen/thrift/gen-cpp/hive_metastore_constants.h 1190490
        trunk/metastore/src/gen/thrift/gen-cpp/hive_metastore_constants.cpp 1190490
        trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Constants.java 1190490
        trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ArchiveUtils.java PRE-CREATION
        trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1190490
        trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1190490
        trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java 1190490
        trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/DummyPartition.java 1190490
        trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 1190490
        trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 1190490
        trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 1195116
        trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1190490
        trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1190490
        trunk/ql/src/test/queries/clientnegative/archive_insert1.q PRE-CREATION
        trunk/ql/src/test/queries/clientnegative/archive_insert2.q PRE-CREATION
        trunk/ql/src/test/queries/clientnegative/archive_insert3.q PRE-CREATION
        trunk/ql/src/test/queries/clientnegative/archive_insert4.q PRE-CREATION
        trunk/ql/src/test/queries/clientnegative/archive_multi1.q PRE-CREATION
        trunk/ql/src/test/queries/clientnegative/archive_multi2.q PRE-CREATION
        trunk/ql/src/test/queries/clientnegative/archive_multi3.q PRE-CREATION
        trunk/ql/src/test/queries/clientnegative/archive_multi4.q PRE-CREATION
        trunk/ql/src/test/queries/clientnegative/archive_multi5.q PRE-CREATION
        trunk/ql/src/test/queries/clientnegative/archive_multi6.q PRE-CREATION
        trunk/ql/src/test/queries/clientnegative/archive_multi7.q PRE-CREATION
        trunk/ql/src/test/queries/clientnegative/archive_partspec1.q PRE-CREATION
        trunk/ql/src/test/queries/clientnegative/archive_partspec2.q PRE-CREATION
        trunk/ql/src/test/queries/clientnegative/archive_partspec3.q PRE-CREATION
        trunk/ql/src/test/queries/clientnegative/archive_partspec4.q PRE-CREATION
        trunk/ql/src/test/queries/clientnegative/archive_partspec5.q PRE-CREATION
        trunk/ql/src/test/queries/clientpositive/archive_corrupt.q PRE-CREATION
        trunk/ql/src/test/queries/clientpositive/archive_multi.q PRE-CREATION
        trunk/ql/src/test/results/clientnegative/archive1.q.out 1190490
        trunk/ql/src/test/results/clientnegative/archive2.q.out 1190490
        trunk/ql/src/test/results/clientnegative/archive_insert1.q.out PRE-CREATION
        trunk/ql/src/test/results/clientnegative/archive_insert2.q.out PRE-CREATION
        trunk/ql/src/test/results/clientnegative/archive_insert3.q.out PRE-CREATION
        trunk/ql/src/test/results/clientnegative/archive_insert4.q.out PRE-CREATION
        trunk/ql/src/test/results/clientnegative/archive_multi1.q.out PRE-CREATION
        trunk/ql/src/test/results/clientnegative/archive_multi2.q.out PRE-CREATION
        trunk/ql/src/test/results/clientnegative/archive_multi3.q.out PRE-CREATION
        trunk/ql/src/test/results/clientnegative/archive_multi4.q.out PRE-CREATION
        trunk/ql/src/test/results/clientnegative/archive_multi5.q.out PRE-CREATION
        trunk/ql/src/test/results/clientnegative/archive_multi6.q.out PRE-CREATION
        trunk/ql/src/test/results/clientnegative/archive_multi7.q.out PRE-CREATION
        trunk/ql/src/test/results/clientnegative/archive_partspec1.q.out PRE-CREATION
        trunk/ql/src/test/results/clientnegative/archive_partspec2.q.out PRE-CREATION
        trunk/ql/src/test/results/clientnegative/archive_partspec3.q.out PRE-CREATION
        trunk/ql/src/test/results/clientnegative/archive_partspec4.q.out PRE-CREATION
        trunk/ql/src/test/results/clientnegative/archive_partspec5.q.out PRE-CREATION
        trunk/ql/src/test/results/clientpositive/archive_corrupt.q.out PRE-CREATION
        trunk/ql/src/test/results/clientpositive/archive_multi.q.out PRE-CREATION

        Diff: https://reviews.apache.org/r/2595/diff

        Testing
        -------

        Support archiving for multiple partitions if the table is partitioned by multiple columns

        Thanks,

        Yongqiang

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2595/ ----------------------------------------------------------- (Updated 2011-10-30 07:34:36.866186) Review request for hive and namit jain. Changes ------- Address review comments except the last one: "Why have a for loop - you are returning inside if anyway the first time ?" I think it is safe to return even for the first time. For any partition returned, it should either be archived or not. If the archive is based on its parent partition col, the partition should also be marked as archived, otherwise, it should not be there. Summary ------- Support archiving for multiple partitions if the table is partitioned by multiple columns This addresses bug HIVE-2278 . https://issues.apache.org/jira/browse/HIVE-2278 Diffs (updated) trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1190490 trunk/metastore/src/gen/thrift/gen-rb/hive_metastore_constants.rb 1190490 trunk/metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_constants.php 1190490 trunk/metastore/src/gen/thrift/gen-py/hive_metastore/constants.py 1190490 trunk/data/files/archive_corrupt.rc UNKNOWN trunk/metastore/if/hive_metastore.thrift 1190490 trunk/metastore/src/gen/thrift/gen-cpp/hive_metastore_constants.h 1190490 trunk/metastore/src/gen/thrift/gen-cpp/hive_metastore_constants.cpp 1190490 trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Constants.java 1190490 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ArchiveUtils.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1190490 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1190490 trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java 1190490 trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/DummyPartition.java 1190490 trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 1190490 trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 1190490 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 1195116 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1190490 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1190490 trunk/ql/src/test/queries/clientnegative/archive_insert1.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_insert2.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_insert3.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_insert4.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_multi1.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_multi2.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_multi3.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_multi4.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_multi5.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_multi6.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_multi7.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_partspec1.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_partspec2.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_partspec3.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_partspec4.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/archive_partspec5.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/archive_corrupt.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/archive_multi.q PRE-CREATION trunk/ql/src/test/results/clientnegative/archive1.q.out 1190490 trunk/ql/src/test/results/clientnegative/archive2.q.out 1190490 trunk/ql/src/test/results/clientnegative/archive_insert1.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_insert2.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_insert3.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_insert4.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_multi1.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_multi2.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_multi3.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_multi4.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_multi5.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_multi6.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_multi7.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_partspec1.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_partspec2.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_partspec3.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_partspec4.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/archive_partspec5.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/archive_corrupt.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/archive_multi.q.out PRE-CREATION Diff: https://reviews.apache.org/r/2595/diff Testing ------- Support archiving for multiple partitions if the table is partitioned by multiple columns Thanks, Yongqiang
        Hide
        Namit Jain added a comment -

        +1

        looks good - will get it in once HIVE-1003 is in, dont want to conflict 2 big patches

        Show
        Namit Jain added a comment - +1 looks good - will get it in once HIVE-1003 is in, dont want to conflict 2 big patches
        Hide
        Namit Jain added a comment -

        ? ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java.orig
        ? ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java.orig
        ? ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java.orig

        Getting merge conflicts, can you resolve them and resubmit the patch.

        Show
        Namit Jain added a comment - ? ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java.orig ? ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java.orig ? ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java.orig Getting merge conflicts, can you resolve them and resubmit the patch.
        Hide
        He Yongqiang added a comment -

        resolved conflicts

        Show
        He Yongqiang added a comment - resolved conflicts
        Hide
        Namit Jain added a comment -

        +1

        running tests

        Show
        Namit Jain added a comment - +1 running tests
        Hide
        Namit Jain added a comment -

        The following tests failed:

        TestMetaStoreEventListener
        TestRemoteHiveMetaStore
        TestCliDriver:
        archive_corrupt
        archive_multi
        authorization_2
        create_view_partitioned
        drop_multi_partitions
        escape1
        inputddl6

        Show
        Namit Jain added a comment - The following tests failed: TestMetaStoreEventListener TestRemoteHiveMetaStore TestCliDriver: archive_corrupt archive_multi authorization_2 create_view_partitioned drop_multi_partitions escape1 inputddl6
        Hide
        He Yongqiang added a comment -

        Some index qfiles are failing for me, and they are failing even without this patch.

        Show
        He Yongqiang added a comment - Some index qfiles are failing for me, and they are failing even without this patch.
        Hide
        Namit Jain added a comment -

        What was the problem ? Running tests again

        Show
        Namit Jain added a comment - What was the problem ? Running tests again
        Hide
        He Yongqiang added a comment -

        fixed some .q.out diffs

        Show
        He Yongqiang added a comment - fixed some .q.out diffs
        Hide
        Namit Jain added a comment -

        +1

        Show
        Namit Jain added a comment - +1
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-h0.21 #1057 (See https://builds.apache.org/job/Hive-trunk-h0.21/1057/)
        HIVE-2278 Support archiving for multiple partitions if the table is partitioned
        by multiple columns (Marcin Kurczych and Yongqiang He via namit)

        namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1197400
        Files :

        • /hive/trunk/data/files/archive_corrupt.rc
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ArchiveUtils.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/DummyPartition.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
        • /hive/trunk/ql/src/test/queries/clientnegative/archive_insert1.q
        • /hive/trunk/ql/src/test/queries/clientnegative/archive_insert2.q
        • /hive/trunk/ql/src/test/queries/clientnegative/archive_insert3.q
        • /hive/trunk/ql/src/test/queries/clientnegative/archive_insert4.q
        • /hive/trunk/ql/src/test/queries/clientnegative/archive_multi1.q
        • /hive/trunk/ql/src/test/queries/clientnegative/archive_multi2.q
        • /hive/trunk/ql/src/test/queries/clientnegative/archive_multi3.q
        • /hive/trunk/ql/src/test/queries/clientnegative/archive_multi4.q
        • /hive/trunk/ql/src/test/queries/clientnegative/archive_multi5.q
        • /hive/trunk/ql/src/test/queries/clientnegative/archive_multi6.q
        • /hive/trunk/ql/src/test/queries/clientnegative/archive_multi7.q
        • /hive/trunk/ql/src/test/queries/clientnegative/archive_partspec1.q
        • /hive/trunk/ql/src/test/queries/clientnegative/archive_partspec2.q
        • /hive/trunk/ql/src/test/queries/clientnegative/archive_partspec3.q
        • /hive/trunk/ql/src/test/queries/clientnegative/archive_partspec4.q
        • /hive/trunk/ql/src/test/queries/clientnegative/archive_partspec5.q
        • /hive/trunk/ql/src/test/queries/clientpositive/archive_corrupt.q
        • /hive/trunk/ql/src/test/queries/clientpositive/archive_multi.q
        • /hive/trunk/ql/src/test/results/clientnegative/archive1.q.out
        • /hive/trunk/ql/src/test/results/clientnegative/archive2.q.out
        • /hive/trunk/ql/src/test/results/clientnegative/archive_insert1.q.out
        • /hive/trunk/ql/src/test/results/clientnegative/archive_insert2.q.out
        • /hive/trunk/ql/src/test/results/clientnegative/archive_insert3.q.out
        • /hive/trunk/ql/src/test/results/clientnegative/archive_insert4.q.out
        • /hive/trunk/ql/src/test/results/clientnegative/archive_multi1.q.out
        • /hive/trunk/ql/src/test/results/clientnegative/archive_multi2.q.out
        • /hive/trunk/ql/src/test/results/clientnegative/archive_multi3.q.out
        • /hive/trunk/ql/src/test/results/clientnegative/archive_multi4.q.out
        • /hive/trunk/ql/src/test/results/clientnegative/archive_multi5.q.out
        • /hive/trunk/ql/src/test/results/clientnegative/archive_multi6.q.out
        • /hive/trunk/ql/src/test/results/clientnegative/archive_multi7.q.out
        • /hive/trunk/ql/src/test/results/clientnegative/archive_partspec1.q.out
        • /hive/trunk/ql/src/test/results/clientnegative/archive_partspec2.q.out
        • /hive/trunk/ql/src/test/results/clientnegative/archive_partspec3.q.out
        • /hive/trunk/ql/src/test/results/clientnegative/archive_partspec4.q.out
        • /hive/trunk/ql/src/test/results/clientnegative/archive_partspec5.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/archive_corrupt.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/archive_multi.q.out
        Show
        Hudson added a comment - Integrated in Hive-trunk-h0.21 #1057 (See https://builds.apache.org/job/Hive-trunk-h0.21/1057/ ) HIVE-2278 Support archiving for multiple partitions if the table is partitioned by multiple columns (Marcin Kurczych and Yongqiang He via namit) namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1197400 Files : /hive/trunk/data/files/archive_corrupt.rc /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ArchiveUtils.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/DummyPartition.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java /hive/trunk/ql/src/test/queries/clientnegative/archive_insert1.q /hive/trunk/ql/src/test/queries/clientnegative/archive_insert2.q /hive/trunk/ql/src/test/queries/clientnegative/archive_insert3.q /hive/trunk/ql/src/test/queries/clientnegative/archive_insert4.q /hive/trunk/ql/src/test/queries/clientnegative/archive_multi1.q /hive/trunk/ql/src/test/queries/clientnegative/archive_multi2.q /hive/trunk/ql/src/test/queries/clientnegative/archive_multi3.q /hive/trunk/ql/src/test/queries/clientnegative/archive_multi4.q /hive/trunk/ql/src/test/queries/clientnegative/archive_multi5.q /hive/trunk/ql/src/test/queries/clientnegative/archive_multi6.q /hive/trunk/ql/src/test/queries/clientnegative/archive_multi7.q /hive/trunk/ql/src/test/queries/clientnegative/archive_partspec1.q /hive/trunk/ql/src/test/queries/clientnegative/archive_partspec2.q /hive/trunk/ql/src/test/queries/clientnegative/archive_partspec3.q /hive/trunk/ql/src/test/queries/clientnegative/archive_partspec4.q /hive/trunk/ql/src/test/queries/clientnegative/archive_partspec5.q /hive/trunk/ql/src/test/queries/clientpositive/archive_corrupt.q /hive/trunk/ql/src/test/queries/clientpositive/archive_multi.q /hive/trunk/ql/src/test/results/clientnegative/archive1.q.out /hive/trunk/ql/src/test/results/clientnegative/archive2.q.out /hive/trunk/ql/src/test/results/clientnegative/archive_insert1.q.out /hive/trunk/ql/src/test/results/clientnegative/archive_insert2.q.out /hive/trunk/ql/src/test/results/clientnegative/archive_insert3.q.out /hive/trunk/ql/src/test/results/clientnegative/archive_insert4.q.out /hive/trunk/ql/src/test/results/clientnegative/archive_multi1.q.out /hive/trunk/ql/src/test/results/clientnegative/archive_multi2.q.out /hive/trunk/ql/src/test/results/clientnegative/archive_multi3.q.out /hive/trunk/ql/src/test/results/clientnegative/archive_multi4.q.out /hive/trunk/ql/src/test/results/clientnegative/archive_multi5.q.out /hive/trunk/ql/src/test/results/clientnegative/archive_multi6.q.out /hive/trunk/ql/src/test/results/clientnegative/archive_multi7.q.out /hive/trunk/ql/src/test/results/clientnegative/archive_partspec1.q.out /hive/trunk/ql/src/test/results/clientnegative/archive_partspec2.q.out /hive/trunk/ql/src/test/results/clientnegative/archive_partspec3.q.out /hive/trunk/ql/src/test/results/clientnegative/archive_partspec4.q.out /hive/trunk/ql/src/test/results/clientnegative/archive_partspec5.q.out /hive/trunk/ql/src/test/results/clientpositive/archive_corrupt.q.out /hive/trunk/ql/src/test/results/clientpositive/archive_multi.q.out
        Hide
        Namit Jain added a comment -

        Committed. Thanks Marcin and Yongqiang

        Show
        Namit Jain added a comment - Committed. Thanks Marcin and Yongqiang
        Hide
        Zhenxiao Luo added a comment -

        Hi, May I ask a question about archive_corrupt.rc?

        As my understanding, this is an RCFile, with corrupted content.

        In archive_corrupt.q, it is trying to load this file into table tstsrcpart.

        Is this expected to be succeeded? Or, we should expect a failure like:

        Failed with exception Wrong file format. Please check the file's format.
        FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask

        Specifically, what is inside archive_corrupt.rc? How is it corrupted?

        Thanks
        Zhenxiao

        Show
        Zhenxiao Luo added a comment - Hi, May I ask a question about archive_corrupt.rc? As my understanding, this is an RCFile, with corrupted content. In archive_corrupt.q, it is trying to load this file into table tstsrcpart. Is this expected to be succeeded? Or, we should expect a failure like: Failed with exception Wrong file format. Please check the file's format. FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask Specifically, what is inside archive_corrupt.rc? How is it corrupted? Thanks Zhenxiao

          People

          • Assignee:
            Marcin Kurczych
            Reporter:
            Namit Jain
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development