Hive
  1. Hive
  2. HIVE-2907

Hive error when dropping a table with large number of partitions

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.9.0
    • Fix Version/s: 0.10.0
    • Component/s: Metastore
    • Labels:
      None
    • Environment:

      General. Hive Metastore bug.

      Description

      Running into an "Out Of Memory" error when trying to drop a table with 128K partitions.

      The methods dropTable in metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
      and dropTable in ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java encounter out of memory errors
      when dropping tables with lots of partitions because they try to load the metadata for every partition into memory.

      1. HIVE-2907.3.patch.txt
        32 kB
        Mousom Dhar Gupta
      2. HIVE-2907.2.patch.txt
        32 kB
        Mousom Dhar Gupta
      3. HIVE-2907.1.patch.txt
        28 kB
        Mousom Dhar Gupta
      4. ASF.LICENSE.NOT.GRANTED--HIVE-2907.D2505.7.patch
        32 kB
        Phabricator
      5. ASF.LICENSE.NOT.GRANTED--HIVE-2907.D2505.6.patch
        32 kB
        Phabricator
      6. ASF.LICENSE.NOT.GRANTED--HIVE-2907.D2505.5.patch
        31 kB
        Phabricator
      7. ASF.LICENSE.NOT.GRANTED--HIVE-2907.D2505.4.patch
        31 kB
        Phabricator
      8. ASF.LICENSE.NOT.GRANTED--HIVE-2907.D2505.3.patch
        30 kB
        Phabricator
      9. ASF.LICENSE.NOT.GRANTED--HIVE-2907.D2505.2.patch
        30 kB
        Phabricator
      10. ASF.LICENSE.NOT.GRANTED--HIVE-2907.D2505.1.patch
        28 kB
        Phabricator

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/)
        HIVE-2907. Hive error when dropping a table with large number of partitions. (mousom via kevinwilfong) (Revision 1311850)

        Result = ABORTED
        kevinwilfong : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311850
        Files :

        • /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
        • /hive/trunk/conf/hive-default.xml.template
        • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
        • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
        • /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
        • /hive/trunk/ql/src/test/queries/clientnegative/drop_table_failure3.q
        • /hive/trunk/ql/src/test/queries/clientpositive/drop_table2.q
        • /hive/trunk/ql/src/test/results/clientnegative/drop_table_failure3.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/drop_table2.q.out
        Show
        Hudson added a comment - Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/ ) HIVE-2907 . Hive error when dropping a table with large number of partitions. (mousom via kevinwilfong) (Revision 1311850) Result = ABORTED kevinwilfong : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311850 Files : /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java /hive/trunk/conf/hive-default.xml.template /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java /hive/trunk/ql/src/test/queries/clientnegative/drop_table_failure3.q /hive/trunk/ql/src/test/queries/clientpositive/drop_table2.q /hive/trunk/ql/src/test/results/clientnegative/drop_table_failure3.q.out /hive/trunk/ql/src/test/results/clientpositive/drop_table2.q.out
        Hide
        Srinivas added a comment -

        Thanks for the timely clarification.

        Show
        Srinivas added a comment - Thanks for the timely clarification.
        Hide
        Carl Steinbach added a comment -

        @Srinivas: This patch was not backported to 0.9.0. The previous fixVersion was inaccurate. I have updated the field to make it clear that this patch will appear in 0.10.0. Thanks.

        Show
        Carl Steinbach added a comment - @Srinivas: This patch was not backported to 0.9.0. The previous fixVersion was inaccurate. I have updated the field to make it clear that this patch will appear in 0.10.0. Thanks.
        Hide
        Srinivas added a comment -

        Also, the method dropTable in "DDLTask.java" appears to be missing the fix proposed in the JIRA. I am looking at the Hive-0.9.0 source code base.

        Show
        Srinivas added a comment - Also, the method dropTable in "DDLTask.java" appears to be missing the fix proposed in the JIRA. I am looking at the Hive-0.9.0 source code base.
        Hide
        Srinivas added a comment -

        I downloaded the source code for Hive-0.9.1. However, it appears that "ObjectStore.java" is missing the fix that fetches partition-metadata in batches. So, it can still cause issues when trying to drop a table with a large number of partitions.

        Proposed fix in method dropTable from ObjectStore.java
        ==========================================================
        int partitionBatchSize = HiveConf.getIntVar(getConf(),
        ConfVars.METASTORE_BATCH_RETRIEVE_MAX);

        // call dropPartition on each of the table's partitions to follow the
        // procedure for cleanly dropping partitions.
        List<MPartition> partsToDelete = listMPartitions(dbName, tableName, partitionBatchSize);
        while (true){
        if (partsToDelete != null || partsToDelete.isEmpty())

        { break; }

        for (MPartition mpart : partsToDelete)

        { dropPartitionCommon(mpart); }

        }

        Show
        Srinivas added a comment - I downloaded the source code for Hive-0.9.1. However, it appears that "ObjectStore.java" is missing the fix that fetches partition-metadata in batches. So, it can still cause issues when trying to drop a table with a large number of partitions. Proposed fix in method dropTable from ObjectStore.java ========================================================== int partitionBatchSize = HiveConf.getIntVar(getConf(), ConfVars.METASTORE_BATCH_RETRIEVE_MAX); // call dropPartition on each of the table's partitions to follow the // procedure for cleanly dropping partitions. List<MPartition> partsToDelete = listMPartitions(dbName, tableName, partitionBatchSize); while (true){ if (partsToDelete != null || partsToDelete.isEmpty()) { break; } for (MPartition mpart : partsToDelete) { dropPartitionCommon(mpart); } }
        Hide
        Phabricator added a comment -

        kevinwilfong has accepted the revision "HIVE-2907 [jira] Hive error when dropping a table with large number of partitions".

        REVISION DETAIL
        https://reviews.facebook.net/D2505

        BRANCH
        svn

        To: JIRA, kevinwilfong, mousom

        Show
        Phabricator added a comment - kevinwilfong has accepted the revision " HIVE-2907 [jira] Hive error when dropping a table with large number of partitions". REVISION DETAIL https://reviews.facebook.net/D2505 BRANCH svn To: JIRA, kevinwilfong, mousom
        Hide
        Ashutosh Chauhan added a comment -

        This issue is closed now. It was released with the fix in 0.9.0. If there is a problem, please open a new jira and link this one with that.

        Show
        Ashutosh Chauhan added a comment - This issue is closed now. It was released with the fix in 0.9.0. If there is a problem, please open a new jira and link this one with that.
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-h0.21 #1367 (See https://builds.apache.org/job/Hive-trunk-h0.21/1367/)
        HIVE-2907. Hive error when dropping a table with large number of partitions. (mousom via kevinwilfong) (Revision 1311850)

        Result = FAILURE
        kevinwilfong : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311850
        Files :

        • /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
        • /hive/trunk/conf/hive-default.xml.template
        • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
        • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
        • /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
        • /hive/trunk/ql/src/test/queries/clientnegative/drop_table_failure3.q
        • /hive/trunk/ql/src/test/queries/clientpositive/drop_table2.q
        • /hive/trunk/ql/src/test/results/clientnegative/drop_table_failure3.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/drop_table2.q.out
        Show
        Hudson added a comment - Integrated in Hive-trunk-h0.21 #1367 (See https://builds.apache.org/job/Hive-trunk-h0.21/1367/ ) HIVE-2907 . Hive error when dropping a table with large number of partitions. (mousom via kevinwilfong) (Revision 1311850) Result = FAILURE kevinwilfong : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311850 Files : /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java /hive/trunk/conf/hive-default.xml.template /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java /hive/trunk/ql/src/test/queries/clientnegative/drop_table_failure3.q /hive/trunk/ql/src/test/queries/clientpositive/drop_table2.q /hive/trunk/ql/src/test/results/clientnegative/drop_table_failure3.q.out /hive/trunk/ql/src/test/results/clientpositive/drop_table2.q.out
        Hide
        Kevin Wilfong added a comment -

        Committed. Thanks Mousom.

        Show
        Kevin Wilfong added a comment - Committed. Thanks Mousom.
        Hide
        Kevin Wilfong added a comment -

        +1 tests passed, will commit.

        Show
        Kevin Wilfong added a comment - +1 tests passed, will commit.
        Hide
        Phabricator added a comment -

        mousom updated the revision "HIVE-2907 [jira] Hive error when dropping a table with large number of partitions".
        Reviewers: JIRA, kevinwilfong

        REVISION DETAIL
        https://reviews.facebook.net/D2505

        AFFECTED FILES
        conf/hive-default.xml.template
        metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java
        metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
        metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
        common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
        ql/src/test/results/clientnegative/drop_table_failure3.q.out
        ql/src/test/results/clientpositive/drop_table2.q.out
        ql/src/test/queries/clientnegative/drop_table_failure3.q
        ql/src/test/queries/clientpositive/drop_table2.q
        ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java

        Show
        Phabricator added a comment - mousom updated the revision " HIVE-2907 [jira] Hive error when dropping a table with large number of partitions". Reviewers: JIRA, kevinwilfong REVISION DETAIL https://reviews.facebook.net/D2505 AFFECTED FILES conf/hive-default.xml.template metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java common/src/java/org/apache/hadoop/hive/conf/HiveConf.java ql/src/test/results/clientnegative/drop_table_failure3.q.out ql/src/test/results/clientpositive/drop_table2.q.out ql/src/test/queries/clientnegative/drop_table_failure3.q ql/src/test/queries/clientpositive/drop_table2.q ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
        Hide
        Mousom Dhar Gupta added a comment -

        Here is the new patch.

        Show
        Mousom Dhar Gupta added a comment - Here is the new patch.
        Hide
        Phabricator added a comment -

        kevinwilfong has commented on the revision "HIVE-2907 [jira] Hive error when dropping a table with large number of partitions".

        I'm getting conflicts trying to apply the patch, could you update your checkout and post an updated diff.

        REVISION DETAIL
        https://reviews.facebook.net/D2505

        Show
        Phabricator added a comment - kevinwilfong has commented on the revision " HIVE-2907 [jira] Hive error when dropping a table with large number of partitions". I'm getting conflicts trying to apply the patch, could you update your checkout and post an updated diff. REVISION DETAIL https://reviews.facebook.net/D2505
        Hide
        Phabricator added a comment -

        mousom updated the revision "HIVE-2907 [jira] Hive error when dropping a table with large number of partitions".
        Reviewers: JIRA, kevinwilfong

        Fixed drop_table_failure3.q

        REVISION DETAIL
        https://reviews.facebook.net/D2505

        AFFECTED FILES
        conf/hive-default.xml.template
        metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java
        metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
        metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
        common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
        ql/src/test/results/clientnegative/drop_table_failure3.q.out
        ql/src/test/results/clientpositive/drop_table2.q.out
        ql/src/test/queries/clientnegative/drop_table_failure3.q
        ql/src/test/queries/clientpositive/drop_table2.q
        ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java

        Show
        Phabricator added a comment - mousom updated the revision " HIVE-2907 [jira] Hive error when dropping a table with large number of partitions". Reviewers: JIRA, kevinwilfong Fixed drop_table_failure3.q REVISION DETAIL https://reviews.facebook.net/D2505 AFFECTED FILES conf/hive-default.xml.template metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java common/src/java/org/apache/hadoop/hive/conf/HiveConf.java ql/src/test/results/clientnegative/drop_table_failure3.q.out ql/src/test/results/clientpositive/drop_table2.q.out ql/src/test/queries/clientnegative/drop_table_failure3.q ql/src/test/queries/clientpositive/drop_table2.q ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
        Hide
        Phabricator added a comment -

        kevinwilfong has requested changes to the revision "HIVE-2907 [jira] Hive error when dropping a table with large number of partitions".

        I'm seeing a lot of test failures in TestNegativeCliDriver. They seem to work ok when run individually, but when I run ant test or ant test -Dtestcase=TestNegativeCliDriver they fail. Could you take a look?

        REVISION DETAIL
        https://reviews.facebook.net/D2505

        BRANCH
        svn

        Show
        Phabricator added a comment - kevinwilfong has requested changes to the revision " HIVE-2907 [jira] Hive error when dropping a table with large number of partitions". I'm seeing a lot of test failures in TestNegativeCliDriver. They seem to work ok when run individually, but when I run ant test or ant test -Dtestcase=TestNegativeCliDriver they fail. Could you take a look? REVISION DETAIL https://reviews.facebook.net/D2505 BRANCH svn
        Hide
        Phabricator added a comment -

        kevinwilfong has accepted the revision "HIVE-2907 [jira] Hive error when dropping a table with large number of partitions".

        +1 Will commit if tests pass.

        REVISION DETAIL
        https://reviews.facebook.net/D2505

        BRANCH
        svn

        Show
        Phabricator added a comment - kevinwilfong has accepted the revision " HIVE-2907 [jira] Hive error when dropping a table with large number of partitions". +1 Will commit if tests pass. REVISION DETAIL https://reviews.facebook.net/D2505 BRANCH svn
        Hide
        Phabricator added a comment -

        mousom updated the revision "HIVE-2907 [jira] Hive error when dropping a table with large number of partitions".
        Reviewers: JIRA, kevinwilfong

        Fixed typo. reaturn => return

        REVISION DETAIL
        https://reviews.facebook.net/D2505

        AFFECTED FILES
        metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java
        metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
        metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
        common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
        ql/src/test/results/clientnegative/drop_table_failure3.q.out
        ql/src/test/results/clientpositive/drop_table2.q.out
        ql/src/test/queries/clientnegative/drop_table_failure3.q
        ql/src/test/queries/clientpositive/drop_table2.q
        ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java

        Show
        Phabricator added a comment - mousom updated the revision " HIVE-2907 [jira] Hive error when dropping a table with large number of partitions". Reviewers: JIRA, kevinwilfong Fixed typo. reaturn => return REVISION DETAIL https://reviews.facebook.net/D2505 AFFECTED FILES metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java common/src/java/org/apache/hadoop/hive/conf/HiveConf.java ql/src/test/results/clientnegative/drop_table_failure3.q.out ql/src/test/results/clientpositive/drop_table2.q.out ql/src/test/queries/clientnegative/drop_table_failure3.q ql/src/test/queries/clientpositive/drop_table2.q ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
        Hide
        Phabricator added a comment -

        kevinwilfong has commented on the revision "HIVE-2907 [jira] Hive error when dropping a table with large number of partitions".

        INLINE COMMENTS
        metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:464-465 Why are you checking for values.size() / 2 here? Admittedly, this works, but shouldn't it be just 2? Or shouldn't you use values.size() / 2 in line 461?
        metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:463 sp returned
        metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:506-507 Same here re values.size() / 2 vs. 2
        metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:505 same here re returned sp
        metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:512 sp returned
        metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:470 sp returned

        REVISION DETAIL
        https://reviews.facebook.net/D2505

        Show
        Phabricator added a comment - kevinwilfong has commented on the revision " HIVE-2907 [jira] Hive error when dropping a table with large number of partitions". INLINE COMMENTS metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:464-465 Why are you checking for values.size() / 2 here? Admittedly, this works, but shouldn't it be just 2? Or shouldn't you use values.size() / 2 in line 461? metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:463 sp returned metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:506-507 Same here re values.size() / 2 vs. 2 metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:505 same here re returned sp metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:512 sp returned metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:470 sp returned REVISION DETAIL https://reviews.facebook.net/D2505
        Hide
        Phabricator added a comment -

        mousom updated the revision "HIVE-2907 [jira] Hive error when dropping a table with large number of partitions".
        Reviewers: JIRA, kevinwilfong

        Updated drop_table_failure3.q by setting
        hive.metastore.batch.retrieve.max=1 so that it makes
        sure that the code is not looping over the same partition to check for NO_DROP.

        REVISION DETAIL
        https://reviews.facebook.net/D2505

        AFFECTED FILES
        metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java
        metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
        metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
        common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
        ql/src/test/results/clientnegative/drop_table_failure3.q.out
        ql/src/test/results/clientpositive/drop_table2.q.out
        ql/src/test/queries/clientnegative/drop_table_failure3.q
        ql/src/test/queries/clientpositive/drop_table2.q
        ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java

        Show
        Phabricator added a comment - mousom updated the revision " HIVE-2907 [jira] Hive error when dropping a table with large number of partitions". Reviewers: JIRA, kevinwilfong Updated drop_table_failure3.q by setting hive.metastore.batch.retrieve.max=1 so that it makes sure that the code is not looping over the same partition to check for NO_DROP. REVISION DETAIL https://reviews.facebook.net/D2505 AFFECTED FILES metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java common/src/java/org/apache/hadoop/hive/conf/HiveConf.java ql/src/test/results/clientnegative/drop_table_failure3.q.out ql/src/test/results/clientpositive/drop_table2.q.out ql/src/test/queries/clientnegative/drop_table_failure3.q ql/src/test/queries/clientpositive/drop_table2.q ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
        Hide
        Phabricator added a comment -

        mousom has commented on the revision "HIVE-2907 [jira] Hive error when dropping a table with large number of partitions".

        May I modify the drop_table_failure3.q test itself to do that? That might serve both purposes.

        INLINE COMMENTS
        metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java:164 Can you please make this a separate task Kevin?

        Thanks
        common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:292 like the *.q file I added?
        metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:544 yup
        metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:1997 my bad. done.
        common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:292 i will add both in *.q files.
        metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java:1399 I thought about it but it's just intuitively does not make sense if we return 'ALL' when 0 is passed.And it might be a very difficult to catch bug for a client.But, on the other hand, you are right that this change might break existing clients and that has to be avoided, so I guess I will make this bug a feature

        REVISION DETAIL
        https://reviews.facebook.net/D2505

        Show
        Phabricator added a comment - mousom has commented on the revision " HIVE-2907 [jira] Hive error when dropping a table with large number of partitions". May I modify the drop_table_failure3.q test itself to do that? That might serve both purposes. INLINE COMMENTS metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java:164 Can you please make this a separate task Kevin? Thanks common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:292 like the *.q file I added? metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:544 yup metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:1997 my bad. done. common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:292 i will add both in *.q files. metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java:1399 I thought about it but it's just intuitively does not make sense if we return 'ALL' when 0 is passed.And it might be a very difficult to catch bug for a client.But, on the other hand, you are right that this change might break existing clients and that has to be avoided, so I guess I will make this bug a feature REVISION DETAIL https://reviews.facebook.net/D2505
        Hide
        Phabricator added a comment -

        kevinwilfong has commented on the revision "HIVE-2907 [jira] Hive error when dropping a table with large number of partitions".

        Could you add a negative test, in which hive.metastore.batch.retrieve.max=1; and otherwise the same as drop_table_failure3.q, this should check that you are not looping over the same partition to check for NO_DROP.

        REVISION DETAIL
        https://reviews.facebook.net/D2505

        Show
        Phabricator added a comment - kevinwilfong has commented on the revision " HIVE-2907 [jira] Hive error when dropping a table with large number of partitions". Could you add a negative test, in which hive.metastore.batch.retrieve.max=1; and otherwise the same as drop_table_failure3.q, this should check that you are not looping over the same partition to check for NO_DROP. REVISION DETAIL https://reviews.facebook.net/D2505
        Hide
        Phabricator added a comment -

        mousom updated the revision "HIVE-2907 [jira] Hive error when dropping a table with large number of partitions".
        Reviewers: JIRA, kevinwilfong

        Fixed drop_table2.q

        REVISION DETAIL
        https://reviews.facebook.net/D2505

        AFFECTED FILES
        metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java
        metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
        metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
        common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
        ql/src/test/results/clientnegative/drop_table_failure3.q.out
        ql/src/test/results/clientpositive/drop_table2.q.out
        ql/src/test/queries/clientnegative/drop_table_failure3.q
        ql/src/test/queries/clientpositive/drop_table2.q
        ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java

        Show
        Phabricator added a comment - mousom updated the revision " HIVE-2907 [jira] Hive error when dropping a table with large number of partitions". Reviewers: JIRA, kevinwilfong Fixed drop_table2.q REVISION DETAIL https://reviews.facebook.net/D2505 AFFECTED FILES metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java common/src/java/org/apache/hadoop/hive/conf/HiveConf.java ql/src/test/results/clientnegative/drop_table_failure3.q.out ql/src/test/results/clientpositive/drop_table2.q.out ql/src/test/queries/clientnegative/drop_table_failure3.q ql/src/test/queries/clientpositive/drop_table2.q ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
        Hide
        Phabricator added a comment -

        mousom updated the revision "HIVE-2907 [jira] Hive error when dropping a table with large number of partitions".
        Reviewers: JIRA, kevinwilfong

        Addressed code review comments.

        REVISION DETAIL
        https://reviews.facebook.net/D2505

        AFFECTED FILES
        metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java
        metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
        metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
        common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
        ql/src/test/results/clientnegative/drop_table_failure3.q.out
        ql/src/test/results/clientpositive/drop_table2.q.out
        ql/src/test/queries/clientnegative/drop_table_failure3.q
        ql/src/test/queries/clientpositive/drop_table2.q
        ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java

        Show
        Phabricator added a comment - mousom updated the revision " HIVE-2907 [jira] Hive error when dropping a table with large number of partitions". Reviewers: JIRA, kevinwilfong Addressed code review comments. REVISION DETAIL https://reviews.facebook.net/D2505 AFFECTED FILES metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java common/src/java/org/apache/hadoop/hive/conf/HiveConf.java ql/src/test/results/clientnegative/drop_table_failure3.q.out ql/src/test/results/clientpositive/drop_table2.q.out ql/src/test/queries/clientnegative/drop_table_failure3.q ql/src/test/queries/clientpositive/drop_table2.q ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
        Hide
        Phabricator added a comment -

        kevinwilfong has commented on the revision "HIVE-2907 [jira] Hive error when dropping a table with large number of partitions".

        This looks really good, thanks for the code clean up, just some small comments.

        INLINE COMMENTS
        metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java:164 It'd be great if we could use the same config here, otherwise alter table will face the same memory issues for tables with lots of partitions.

        This could be a separate task too.
        metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java:1399-1401 If any client code already assumes that if 0 is max, it returns all partitions, which, given the code in HiveAlterHandler, seems possible, this will break that code. Could we treat 0 as infinity as well?
        metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java:1565-1567 Same here regarding 0 => infinity
        metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:1997 Can you remove this line if it's not needed.
        metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:544 Shouldn't this be "table should not have existed"
        common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:292 Could you add a testcase, where you set this really low, e.g. 1, and you drop a table with multiple partitions.

        Also, could you add a similar test where you make one of the partitions NO_DROP, and make sure it fails.

        REVISION DETAIL
        https://reviews.facebook.net/D2505

        Show
        Phabricator added a comment - kevinwilfong has commented on the revision " HIVE-2907 [jira] Hive error when dropping a table with large number of partitions". This looks really good, thanks for the code clean up, just some small comments. INLINE COMMENTS metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java:164 It'd be great if we could use the same config here, otherwise alter table will face the same memory issues for tables with lots of partitions. This could be a separate task too. metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java:1399-1401 If any client code already assumes that if 0 is max, it returns all partitions, which, given the code in HiveAlterHandler, seems possible, this will break that code. Could we treat 0 as infinity as well? metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java:1565-1567 Same here regarding 0 => infinity metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:1997 Can you remove this line if it's not needed. metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:544 Shouldn't this be "table should not have existed" common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:292 Could you add a testcase, where you set this really low, e.g. 1, and you drop a table with multiple partitions. Also, could you add a similar test where you make one of the partitions NO_DROP, and make sure it fails. REVISION DETAIL https://reviews.facebook.net/D2505
        Hide
        Phabricator added a comment -

        mousom requested code review of "HIVE-2907 [jira] Hive error when dropping a table with large number of partitions".
        Reviewers: JIRA

        https://issues.apache.org/jira/browse/HIVE-2907

        Hive error when dropping a table with large number of partitions.

        Running into an "Out Of Memory" error when trying to drop a table with 128K partitions.

        The methods dropTable in metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
        and dropTable in ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java encounter out of memory errors
        when dropping tables with lots of partitions because they try to load the metadata for every partition into memory.

        TEST PLAN
        EMPTY

        REVISION DETAIL
        https://reviews.facebook.net/D2505

        AFFECTED FILES
        metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java
        metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
        metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
        common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
        ql/src/test/results/clientnegative/drop_table_failure3.q.out
        ql/src/test/queries/clientnegative/drop_table_failure3.q
        ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java

        MANAGE HERALD DIFFERENTIAL RULES
        https://reviews.facebook.net/herald/view/differential/

        WHY DID I GET THIS EMAIL?
        https://reviews.facebook.net/herald/transcript/5595/

        Tip: use the X-Herald-Rules header to filter Herald messages in your client.

        Show
        Phabricator added a comment - mousom requested code review of " HIVE-2907 [jira] Hive error when dropping a table with large number of partitions". Reviewers: JIRA https://issues.apache.org/jira/browse/HIVE-2907 Hive error when dropping a table with large number of partitions. Running into an "Out Of Memory" error when trying to drop a table with 128K partitions. The methods dropTable in metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java and dropTable in ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java encounter out of memory errors when dropping tables with lots of partitions because they try to load the metadata for every partition into memory. TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D2505 AFFECTED FILES metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java common/src/java/org/apache/hadoop/hive/conf/HiveConf.java ql/src/test/results/clientnegative/drop_table_failure3.q.out ql/src/test/queries/clientnegative/drop_table_failure3.q ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/5595/ Tip: use the X-Herald-Rules header to filter Herald messages in your client.
        Hide
        Mousom Dhar Gupta added a comment -

        For ObjectStore::dropTable am dropping pratitions in batches. And for DDLTask::dropTable am
        getting all partition names and then getting partition metadata from those names in batches.

        The batch size is determined by a config parameter (set to 1000 to start with).

        Thanks

        Show
        Mousom Dhar Gupta added a comment - For ObjectStore::dropTable am dropping pratitions in batches. And for DDLTask::dropTable am getting all partition names and then getting partition metadata from those names in batches. The batch size is determined by a config parameter (set to 1000 to start with). Thanks

          People

          • Assignee:
            Mousom Dhar Gupta
            Reporter:
            Mousom Dhar Gupta
          • Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Time Tracking

              Estimated:
              Original Estimate - 10h
              10h
              Remaining:
              Remaining Estimate - 10h
              10h
              Logged:
              Time Spent - Not Specified
              Not Specified

                Development