Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-4367

enhance TRUNCATE syntex to drop data of external table

    Details

    • Type: Improvement
    • Status: Patch Available
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 0.11.0
    • Fix Version/s: None
    • Component/s: Query Processor
    • Labels:
      None

      Description

      In my use case ,
      sometimes I have to remove data of external tables to free up storage space of the cluster .
      So it's necessary for to enhance the syntax like
      "TRUNCATE TABLE srcpart_truncate PARTITION (dt='201130412') FORCE;"
      to remove data from EXTERNAL table.

      And I add a configuration property to enable remove data to Trash
      <property>
      <name>hive.truncate.skiptrash</name>
      <value>false</value>
      <description>
      if true will remove data to trash, else false drop data immediately
      </description>
      </property>

      For example :
      hive (default)> TRUNCATE TABLE external1 partition (ds='11');
      FAILED: Error in semantic analysis: Cannot truncate non-managed table external1
      hive (default)> TRUNCATE TABLE external1 partition (ds='11') FORCE;
      [2013-04-16 17:15:52]: Compile Start
      [2013-04-16 17:15:52]: Compile End
      [2013-04-16 17:15:52]: OK
      [2013-04-16 17:15:52]: Time taken: 0.413 seconds

      hive (default)> set hive.truncate.skiptrash;
      hive.truncate.skiptrash=false

      hive (default)> set hive.truncate.skiptrash=true;
      hive (default)> TRUNCATE TABLE external1 partition (ds='12') FORCE;
      [2013-04-16 17:16:21]: Compile Start
      [2013-04-16 17:16:21]: Compile End
      [2013-04-16 17:16:21]: OK
      [2013-04-16 17:16:21]: Time taken: 0.143 seconds

      hive (default)> dfs -ls /user/test/.Trash/Current/;
      Found 1 items
      drwxr-xr-x -test supergroup 0 2013-04-16 17:06 /user/test/.Trash/Current/ds=11

      1. HIVE-4367.2.patch.txt
        5 kB
        Navis
      2. HIVE-4367-1.patch
        7 kB
        caofangkun

        Issue Links

          Activity

          Show
          caofangkun caofangkun added a comment - https://reviews.apache.org/r/10600/
          Hide
          teddy.choi Teddy Choi added a comment -

          This issue was unassigned, so I assigned this issue to me. Then caofangkun uploaded a patch for it. I have no right to assign it to him, so I'll make it unassigned. Please assign it to him, other committers.

          I'll review and test this patch.

          Show
          teddy.choi Teddy Choi added a comment - This issue was unassigned, so I assigned this issue to me. Then caofangkun uploaded a patch for it. I have no right to assign it to him, so I'll make it unassigned. Please assign it to him, other committers. I'll review and test this patch.
          Hide
          caofangkun caofangkun added a comment -

          Hi Teddy Choi ,Sorry for that I did not notice you have assigned this issue when I upload the patch. I'm not a committer yet,so please feel free and assign this issue.
          Thank you.

          Show
          caofangkun caofangkun added a comment - Hi Teddy Choi ,Sorry for that I did not notice you have assigned this issue when I upload the patch. I'm not a committer yet,so please feel free and assign this issue. Thank you.
          Hide
          teddy.choi Teddy Choi added a comment -

          It's okay, caofangkun. And thank you for your patch.

          Show
          teddy.choi Teddy Choi added a comment - It's okay, caofangkun . And thank you for your patch.
          Hide
          teddy.choi Teddy Choi added a comment -

          I tested the second patch on review board, but it makes some ANTLR errors. I'm running it again with a log option to see more details.

          Please read the "Contributing your work" on https://cwiki.apache.org/Hive/howtocontribute.html#HowToContribute-Contributingyourwork , caofangkun. It says that you should upload your new patch and click "Submit Patch" to request a review.

          Show
          teddy.choi Teddy Choi added a comment - I tested the second patch on review board, but it makes some ANTLR errors. I'm running it again with a log option to see more details. Please read the "Contributing your work" on https://cwiki.apache.org/Hive/howtocontribute.html#HowToContribute-Contributingyourwork , caofangkun . It says that you should upload your new patch and click "Submit Patch" to request a review.
          Hide
          teddy.choi Teddy Choi added a comment -

          I tested the second patch on review board, and it completed successfully. The bug I found before was due to HIVE-4394, not to this patch.

          Show
          teddy.choi Teddy Choi added a comment - I tested the second patch on review board, and it completed successfully. The bug I found before was due to HIVE-4394 , not to this patch.
          Hide
          caofangkun caofangkun added a comment -

          Thanks Teddy Choi I have read the cwiki , it's helpful for me.

          Show
          caofangkun caofangkun added a comment - Thanks Teddy Choi I have read the cwiki , it's helpful for me.
          Hide
          navis Navis added a comment -

          Rebased to trunk

          Show
          navis Navis added a comment - Rebased to trunk
          Hide
          hiveqa Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12624284/HIVE-4367.2.patch.txt

          ERROR: -1 due to 3 failed/errored test(s), 4943 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_truncate_table_failure1
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_truncate_table_failure3
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_truncate_table_failure4
          

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/989/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/989/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 3 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12624284

          Show
          hiveqa Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12624284/HIVE-4367.2.patch.txt ERROR: -1 due to 3 failed/errored test(s), 4943 tests executed Failed tests: org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_truncate_table_failure1 org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_truncate_table_failure3 org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_truncate_table_failure4 Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/989/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/989/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 3 tests failed This message is automatically generated. ATTACHMENT ID: 12624284
          Hide
          leftylev Lefty Leverenz added a comment -

          When this is ready, documentation will be needed in the wiki:

          Configuration Properties doesn't have a section for DDL. One could be added, or "Query Execution" could be renamed "HiveQL Execution" to encompass DDL and queries. (That would also be useful for hive.support.quoted.identifiers from HIVE-6013, which applies to both queries and DDL.)

          Show
          leftylev Lefty Leverenz added a comment - When this is ready, documentation will be needed in the wiki: syntax goes in DDL's Truncate Table section hive.truncate.skiptrash goes in Configuration Properties Configuration Properties doesn't have a section for DDL. One could be added, or "Query Execution" could be renamed "HiveQL Execution" to encompass DDL and queries. (That would also be useful for hive.support.quoted.identifiers from HIVE-6013 , which applies to both queries and DDL.)
          Hide
          leftylev Lefty Leverenz added a comment -

          Looks like true & false got switched here:

          + <name>hive.truncate.skiptrash</name>
          + <value>false</value>
          + <description>
          + if true will remove data to trash, else false drop data immediately
          + </description>

          Show
          leftylev Lefty Leverenz added a comment - Looks like true & false got switched here: + <name>hive.truncate.skiptrash</name> + <value>false</value> + <description> + if true will remove data to trash, else false drop data immediately + </description>

            People

            • Assignee:
              caofangkun caofangkun
              Reporter:
              caofangkun caofangkun
            • Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:

                Development