Hive
  1. Hive
  2. HIVE-4367

enhance TRUNCATE syntex to drop data of external table

    Details

    • Type: Improvement Improvement
    • Status: Patch Available
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: 0.11.0
    • Fix Version/s: None
    • Component/s: Query Processor
    • Labels:
      None

      Description

      In my use case ,
      sometimes I have to remove data of external tables to free up storage space of the cluster .
      So it's necessary for to enhance the syntax like
      "TRUNCATE TABLE srcpart_truncate PARTITION (dt='201130412') FORCE;"
      to remove data from EXTERNAL table.

      And I add a configuration property to enable remove data to Trash
      <property>
      <name>hive.truncate.skiptrash</name>
      <value>false</value>
      <description>
      if true will remove data to trash, else false drop data immediately
      </description>
      </property>

      For example :
      hive (default)> TRUNCATE TABLE external1 partition (ds='11');
      FAILED: Error in semantic analysis: Cannot truncate non-managed table external1
      hive (default)> TRUNCATE TABLE external1 partition (ds='11') FORCE;
      [2013-04-16 17:15:52]: Compile Start
      [2013-04-16 17:15:52]: Compile End
      [2013-04-16 17:15:52]: OK
      [2013-04-16 17:15:52]: Time taken: 0.413 seconds

      hive (default)> set hive.truncate.skiptrash;
      hive.truncate.skiptrash=false

      hive (default)> set hive.truncate.skiptrash=true;
      hive (default)> TRUNCATE TABLE external1 partition (ds='12') FORCE;
      [2013-04-16 17:16:21]: Compile Start
      [2013-04-16 17:16:21]: Compile End
      [2013-04-16 17:16:21]: OK
      [2013-04-16 17:16:21]: Time taken: 0.143 seconds

      hive (default)> dfs -ls /user/test/.Trash/Current/;
      Found 1 items
      drwxr-xr-x -test supergroup 0 2013-04-16 17:06 /user/test/.Trash/Current/ds=11

      1. HIVE-4367.2.patch.txt
        5 kB
        Navis
      2. HIVE-4367-1.patch
        7 kB
        caofangkun

        Issue Links

          Activity

          Show
          caofangkun added a comment - https://reviews.apache.org/r/10600/
          Hide
          Teddy Choi added a comment -

          This issue was unassigned, so I assigned this issue to me. Then caofangkun uploaded a patch for it. I have no right to assign it to him, so I'll make it unassigned. Please assign it to him, other committers.

          I'll review and test this patch.

          Show
          Teddy Choi added a comment - This issue was unassigned, so I assigned this issue to me. Then caofangkun uploaded a patch for it. I have no right to assign it to him, so I'll make it unassigned. Please assign it to him, other committers. I'll review and test this patch.
          Hide
          caofangkun added a comment -

          Hi Teddy Choi ,Sorry for that I did not notice you have assigned this issue when I upload the patch. I'm not a committer yet,so please feel free and assign this issue.
          Thank you.

          Show
          caofangkun added a comment - Hi Teddy Choi ,Sorry for that I did not notice you have assigned this issue when I upload the patch. I'm not a committer yet,so please feel free and assign this issue. Thank you.
          Hide
          Teddy Choi added a comment -

          It's okay, caofangkun. And thank you for your patch.

          Show
          Teddy Choi added a comment - It's okay, caofangkun . And thank you for your patch.
          Hide
          Teddy Choi added a comment -

          I tested the second patch on review board, but it makes some ANTLR errors. I'm running it again with a log option to see more details.

          Please read the "Contributing your work" on https://cwiki.apache.org/Hive/howtocontribute.html#HowToContribute-Contributingyourwork , caofangkun. It says that you should upload your new patch and click "Submit Patch" to request a review.

          Show
          Teddy Choi added a comment - I tested the second patch on review board, but it makes some ANTLR errors. I'm running it again with a log option to see more details. Please read the "Contributing your work" on https://cwiki.apache.org/Hive/howtocontribute.html#HowToContribute-Contributingyourwork , caofangkun . It says that you should upload your new patch and click "Submit Patch" to request a review.
          Hide
          Teddy Choi added a comment -

          I tested the second patch on review board, and it completed successfully. The bug I found before was due to HIVE-4394, not to this patch.

          Show
          Teddy Choi added a comment - I tested the second patch on review board, and it completed successfully. The bug I found before was due to HIVE-4394 , not to this patch.
          Hide
          caofangkun added a comment -

          Thanks Teddy Choi I have read the cwiki , it's helpful for me.

          Show
          caofangkun added a comment - Thanks Teddy Choi I have read the cwiki , it's helpful for me.
          Hide
          Navis added a comment -

          Rebased to trunk

          Show
          Navis added a comment - Rebased to trunk
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12624284/HIVE-4367.2.patch.txt

          ERROR: -1 due to 3 failed/errored test(s), 4943 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_truncate_table_failure1
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_truncate_table_failure3
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_truncate_table_failure4
          

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/989/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/989/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 3 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12624284

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12624284/HIVE-4367.2.patch.txt ERROR: -1 due to 3 failed/errored test(s), 4943 tests executed Failed tests: org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_truncate_table_failure1 org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_truncate_table_failure3 org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_truncate_table_failure4 Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/989/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/989/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 3 tests failed This message is automatically generated. ATTACHMENT ID: 12624284
          Hide
          Lefty Leverenz added a comment -

          When this is ready, documentation will be needed in the wiki:

          Configuration Properties doesn't have a section for DDL. One could be added, or "Query Execution" could be renamed "HiveQL Execution" to encompass DDL and queries. (That would also be useful for hive.support.quoted.identifiers from HIVE-6013, which applies to both queries and DDL.)

          Show
          Lefty Leverenz added a comment - When this is ready, documentation will be needed in the wiki: syntax goes in DDL's Truncate Table section hive.truncate.skiptrash goes in Configuration Properties Configuration Properties doesn't have a section for DDL. One could be added, or "Query Execution" could be renamed "HiveQL Execution" to encompass DDL and queries. (That would also be useful for hive.support.quoted.identifiers from HIVE-6013 , which applies to both queries and DDL.)
          Hide
          Lefty Leverenz added a comment -

          Looks like true & false got switched here:

          + <name>hive.truncate.skiptrash</name>
          + <value>false</value>
          + <description>
          + if true will remove data to trash, else false drop data immediately
          + </description>

          Show
          Lefty Leverenz added a comment - Looks like true & false got switched here: + <name>hive.truncate.skiptrash</name> + <value>false</value> + <description> + if true will remove data to trash, else false drop data immediately + </description>

            People

            • Assignee:
              caofangkun
              Reporter:
              caofangkun
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:

                Development