Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-6980

Drop table by using direct sql

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.12.0
    • 4.0.0-alpha-1
    • Metastore
    • None

    Description

      Dropping table which has lots of partitions is slow. Even after applying the patch of HIVE-6265, the drop table still takes hours (100K+ partitions).

      The fixes come with two parts:
      1. use directSQL to query the partitions protect mode;
      the current implementation needs to transfer the Partition object to client and check the protect mode for each partition. I'd like to move this part of logic to metastore. The check will be done by direct sql (if direct sql is disabled, execute the same logic in the ObjectStore);

      2. use directSQL to drop partitions for table;
      there maybe two solutions here:
      1. add "DELETE CASCADE" in the schema. In this way we only need to delete entries from partitions table use direct sql. May need to change datanucleus.deletionPolicy = DataNucleus.
      2. clean up the dependent tables by issue DELETE statement. This also needs to turn on datanucleus.query.sql.allowAll

      Both of above solutions should be able to fix the problem. The DELETE CASCADE has to change schemas and prepare upgrade scripts. The second solutions added maintenance cost if new tables added in the future releases.

      Please advice.

      Attachments

        1. drop_table_after.png
          610 kB
          Peter Vary
        2. drop_table_before.png
          615 kB
          Peter Vary
        3. HIVE-6980.2.patch
          26 kB
          Peter Vary
        4. HIVE-6980.3.patch
          26 kB
          Peter Vary
        5. HIVE-6980.4.patch
          23 kB
          Peter Vary
        6. HIVE-6980.5.patch
          23 kB
          Peter Vary
        7. HIVE-6980.6.patch
          23 kB
          Peter Vary
        8. HIVE-6980.7.patch
          23 kB
          Peter Vary
        9. HIVE-6980.patch
          26 kB
          Peter Vary

        Issue Links

          Activity

            People

              pvary Peter Vary
              selinazh Selina Zhang
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: