Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.11.0
    • Component/s: Query Processor
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      truncate the data but leave the table and metadata intact.

      1. HIVE-446.D7371.1.patch
        18 kB
        Phabricator
      2. HIVE-446.D7371.2.patch
        43 kB
        Phabricator
      3. HIVE-446.D7371.3.patch
        48 kB
        Phabricator
      4. HIVE-446.D7371.4.patch
        48 kB
        Phabricator

        Issue Links

          Activity

          Hide
          caofangkun added a comment -

          Thank you Gang Tim Liu
          Many times I have to remove data from external table to free up storage space of the cluster. So it's necessary for me to have some statement like "truncate ... force" to remove data.
          I submited an issue https://issues.apache.org/jira/browse/HIVE-4367
          Just in case it may be some helpful for people have a similar need .

          Show
          caofangkun added a comment - Thank you Gang Tim Liu Many times I have to remove data from external table to free up storage space of the cluster. So it's necessary for me to have some statement like "truncate ... force" to remove data. I submited an issue https://issues.apache.org/jira/browse/HIVE-4367 Just in case it may be some helpful for people have a similar need .
          Hide
          Gang Tim Liu added a comment -

          "External table" is used in the context where data is not fully managed. If it ends up that there is a need to remove data behind external table, a question can be asked "why do you define it as external table?".

          Saying that, possibly the proposed syntax and semantics are not consistent to "external table" use case.

          thanks

          Show
          Gang Tim Liu added a comment - "External table" is used in the context where data is not fully managed. If it ends up that there is a need to remove data behind external table, a question can be asked "why do you define it as external table?". Saying that, possibly the proposed syntax and semantics are not consistent to "external table" use case. thanks
          Hide
          caofangkun added a comment -

          Hi ALL:
          Whether it is necessary to enhance the syntax like
          "TRUNCATE TABLE srcpart_truncate PARTITION (dt='201130412') FORCE;"
          to remove data from EXTERNAL table ?

          Show
          caofangkun added a comment - Hi ALL: Whether it is necessary to enhance the syntax like "TRUNCATE TABLE srcpart_truncate PARTITION (dt='201130412') FORCE;" to remove data from EXTERNAL table ?
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/)
          HIVE-446 Implement TRUNCATE
          (Navis via namit) (Revision 1427681)

          Result = ABORTED
          namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1427681
          Files :

          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/DDLWork.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/TruncateTableDesc.java
          • /hive/trunk/ql/src/test/queries/clientnegative/truncate_table_failure1.q
          • /hive/trunk/ql/src/test/queries/clientnegative/truncate_table_failure2.q
          • /hive/trunk/ql/src/test/queries/clientnegative/truncate_table_failure3.q
          • /hive/trunk/ql/src/test/queries/clientnegative/truncate_table_failure4.q
          • /hive/trunk/ql/src/test/queries/clientpositive/truncate_table.q
          • /hive/trunk/ql/src/test/results/clientnegative/truncate_table_failure1.q.out
          • /hive/trunk/ql/src/test/results/clientnegative/truncate_table_failure2.q.out
          • /hive/trunk/ql/src/test/results/clientnegative/truncate_table_failure3.q.out
          • /hive/trunk/ql/src/test/results/clientnegative/truncate_table_failure4.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/truncate_table.q.out
          Show
          Hudson added a comment - Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/ ) HIVE-446 Implement TRUNCATE (Navis via namit) (Revision 1427681) Result = ABORTED namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1427681 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/DDLWork.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/TruncateTableDesc.java /hive/trunk/ql/src/test/queries/clientnegative/truncate_table_failure1.q /hive/trunk/ql/src/test/queries/clientnegative/truncate_table_failure2.q /hive/trunk/ql/src/test/queries/clientnegative/truncate_table_failure3.q /hive/trunk/ql/src/test/queries/clientnegative/truncate_table_failure4.q /hive/trunk/ql/src/test/queries/clientpositive/truncate_table.q /hive/trunk/ql/src/test/results/clientnegative/truncate_table_failure1.q.out /hive/trunk/ql/src/test/results/clientnegative/truncate_table_failure2.q.out /hive/trunk/ql/src/test/results/clientnegative/truncate_table_failure3.q.out /hive/trunk/ql/src/test/results/clientnegative/truncate_table_failure4.q.out /hive/trunk/ql/src/test/results/clientpositive/truncate_table.q.out
          Hide
          Navis added a comment -

          Mark Grover Thanks!

          Show
          Navis added a comment - Mark Grover Thanks!
          Hide
          Phabricator added a comment -

          mgrover has commented on the revision "HIVE-446 [jira] Implement TRUNCATE".

          I am happy to take care of all rmr changes in HIVE-3701

          REVISION DETAIL
          https://reviews.facebook.net/D7371

          To: JIRA, navis
          Cc: njain, mgrover

          Show
          Phabricator added a comment - mgrover has commented on the revision " HIVE-446 [jira] Implement TRUNCATE". I am happy to take care of all rmr changes in HIVE-3701 REVISION DETAIL https://reviews.facebook.net/D7371 To: JIRA, navis Cc: njain, mgrover
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-h0.21 #1890 (See https://builds.apache.org/job/Hive-trunk-h0.21/1890/)
          HIVE-446 Implement TRUNCATE
          (Navis via namit) (Revision 1427681)

          Result = FAILURE
          namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1427681
          Files :

          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/DDLWork.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/TruncateTableDesc.java
          • /hive/trunk/ql/src/test/queries/clientnegative/truncate_table_failure1.q
          • /hive/trunk/ql/src/test/queries/clientnegative/truncate_table_failure2.q
          • /hive/trunk/ql/src/test/queries/clientnegative/truncate_table_failure3.q
          • /hive/trunk/ql/src/test/queries/clientnegative/truncate_table_failure4.q
          • /hive/trunk/ql/src/test/queries/clientpositive/truncate_table.q
          • /hive/trunk/ql/src/test/results/clientnegative/truncate_table_failure1.q.out
          • /hive/trunk/ql/src/test/results/clientnegative/truncate_table_failure2.q.out
          • /hive/trunk/ql/src/test/results/clientnegative/truncate_table_failure3.q.out
          • /hive/trunk/ql/src/test/results/clientnegative/truncate_table_failure4.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/truncate_table.q.out
          Show
          Hudson added a comment - Integrated in Hive-trunk-h0.21 #1890 (See https://builds.apache.org/job/Hive-trunk-h0.21/1890/ ) HIVE-446 Implement TRUNCATE (Navis via namit) (Revision 1427681) Result = FAILURE namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1427681 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/DDLWork.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/TruncateTableDesc.java /hive/trunk/ql/src/test/queries/clientnegative/truncate_table_failure1.q /hive/trunk/ql/src/test/queries/clientnegative/truncate_table_failure2.q /hive/trunk/ql/src/test/queries/clientnegative/truncate_table_failure3.q /hive/trunk/ql/src/test/queries/clientnegative/truncate_table_failure4.q /hive/trunk/ql/src/test/queries/clientpositive/truncate_table.q /hive/trunk/ql/src/test/results/clientnegative/truncate_table_failure1.q.out /hive/trunk/ql/src/test/results/clientnegative/truncate_table_failure2.q.out /hive/trunk/ql/src/test/results/clientnegative/truncate_table_failure3.q.out /hive/trunk/ql/src/test/results/clientnegative/truncate_table_failure4.q.out /hive/trunk/ql/src/test/results/clientpositive/truncate_table.q.out
          Hide
          Namit Jain added a comment -

          Committed. Thanks Navis

          Show
          Namit Jain added a comment - Committed. Thanks Navis
          Hide
          Namit Jain added a comment -

          It should be part of DDL

          Show
          Namit Jain added a comment - It should be part of DDL
          Hide
          Navis added a comment -

          @Namit Jain,
          done in https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML, but I'm not sure it's DDL or DML.

          Show
          Navis added a comment - @Namit Jain, done in https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML , but I'm not sure it's DDL or DML.
          Hide
          Namit Jain added a comment -

          @Navis, can you add documentation for this ?

          Show
          Namit Jain added a comment - @Navis, can you add documentation for this ?
          Hide
          Namit Jain added a comment -

          +1

          Running tests

          Show
          Namit Jain added a comment - +1 Running tests
          Hide
          Phabricator added a comment -

          navis has commented on the revision "HIVE-446 [jira] Implement TRUNCATE".

          HIVE-3701 seemed good place for it, fixing -rmr cases altogether. Doesn't it?

          INLINE COMMENTS
          ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java:3926 fixed.

          REVISION DETAIL
          https://reviews.facebook.net/D7371

          To: JIRA, navis
          Cc: njain, mgrover

          Show
          Phabricator added a comment - navis has commented on the revision " HIVE-446 [jira] Implement TRUNCATE". HIVE-3701 seemed good place for it, fixing -rmr cases altogether. Doesn't it? INLINE COMMENTS ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java:3926 fixed. REVISION DETAIL https://reviews.facebook.net/D7371 To: JIRA, navis Cc: njain, mgrover
          Hide
          Phabricator added a comment -

          navis updated the revision "HIVE-446 [jira] Implement TRUNCATE".
          Reviewers: JIRA

          Removed redundant output entities

          REVISION DETAIL
          https://reviews.facebook.net/D7371

          AFFECTED FILES
          ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
          ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java
          ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g
          ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java
          ql/src/java/org/apache/hadoop/hive/ql/plan/DDLWork.java
          ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java
          ql/src/java/org/apache/hadoop/hive/ql/plan/TruncateTableDesc.java
          ql/src/test/queries/clientnegative/truncate_table_failure1.q
          ql/src/test/queries/clientnegative/truncate_table_failure2.q
          ql/src/test/queries/clientnegative/truncate_table_failure3.q
          ql/src/test/queries/clientnegative/truncate_table_failure4.q
          ql/src/test/queries/clientpositive/truncate_table.q
          ql/src/test/results/clientnegative/truncate_table_failure1.q.out
          ql/src/test/results/clientnegative/truncate_table_failure2.q.out
          ql/src/test/results/clientnegative/truncate_table_failure3.q.out
          ql/src/test/results/clientnegative/truncate_table_failure4.q.out
          ql/src/test/results/clientpositive/truncate_table.q.out

          To: JIRA, navis
          Cc: njain, mgrover

          Show
          Phabricator added a comment - navis updated the revision " HIVE-446 [jira] Implement TRUNCATE". Reviewers: JIRA Removed redundant output entities REVISION DETAIL https://reviews.facebook.net/D7371 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java ql/src/java/org/apache/hadoop/hive/ql/plan/DDLWork.java ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java ql/src/java/org/apache/hadoop/hive/ql/plan/TruncateTableDesc.java ql/src/test/queries/clientnegative/truncate_table_failure1.q ql/src/test/queries/clientnegative/truncate_table_failure2.q ql/src/test/queries/clientnegative/truncate_table_failure3.q ql/src/test/queries/clientnegative/truncate_table_failure4.q ql/src/test/queries/clientpositive/truncate_table.q ql/src/test/results/clientnegative/truncate_table_failure1.q.out ql/src/test/results/clientnegative/truncate_table_failure2.q.out ql/src/test/results/clientnegative/truncate_table_failure3.q.out ql/src/test/results/clientnegative/truncate_table_failure4.q.out ql/src/test/results/clientpositive/truncate_table.q.out To: JIRA, navis Cc: njain, mgrover
          Hide
          Phabricator added a comment -

          njain has commented on the revision "HIVE-446 [jira] Implement TRUNCATE".

          The changes look good to me.
          Can you file a follow-up jira to add -rmr to hadoop shims ?

          INLINE COMMENTS
          ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java:3926 not a big deal - but, the outputs should be already populated.
          Anyway, it is a set, so it should not matter.

          REVISION DETAIL
          https://reviews.facebook.net/D7371

          To: JIRA, navis
          Cc: njain, mgrover

          Show
          Phabricator added a comment - njain has commented on the revision " HIVE-446 [jira] Implement TRUNCATE". The changes look good to me. Can you file a follow-up jira to add -rmr to hadoop shims ? INLINE COMMENTS ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java:3926 not a big deal - but, the outputs should be already populated. Anyway, it is a set, so it should not matter. REVISION DETAIL https://reviews.facebook.net/D7371 To: JIRA, navis Cc: njain, mgrover
          Hide
          Phabricator added a comment -

          navis updated the revision "HIVE-446 [jira] Implement TRUNCATE".
          Reviewers: JIRA

          Addressed comments

          REVISION DETAIL
          https://reviews.facebook.net/D7371

          AFFECTED FILES
          ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
          ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java
          ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g
          ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java
          ql/src/java/org/apache/hadoop/hive/ql/plan/DDLWork.java
          ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java
          ql/src/java/org/apache/hadoop/hive/ql/plan/TruncateTableDesc.java
          ql/src/test/queries/clientnegative/truncate_table_failure1.q
          ql/src/test/queries/clientnegative/truncate_table_failure2.q
          ql/src/test/queries/clientnegative/truncate_table_failure3.q
          ql/src/test/queries/clientnegative/truncate_table_failure4.q
          ql/src/test/queries/clientpositive/truncate_table.q
          ql/src/test/results/clientnegative/truncate_table_failure1.q.out
          ql/src/test/results/clientnegative/truncate_table_failure2.q.out
          ql/src/test/results/clientnegative/truncate_table_failure3.q.out
          ql/src/test/results/clientnegative/truncate_table_failure4.q.out
          ql/src/test/results/clientpositive/truncate_table.q.out

          To: JIRA, navis
          Cc: njain, mgrover

          Show
          Phabricator added a comment - navis updated the revision " HIVE-446 [jira] Implement TRUNCATE". Reviewers: JIRA Addressed comments REVISION DETAIL https://reviews.facebook.net/D7371 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java ql/src/java/org/apache/hadoop/hive/ql/plan/DDLWork.java ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java ql/src/java/org/apache/hadoop/hive/ql/plan/TruncateTableDesc.java ql/src/test/queries/clientnegative/truncate_table_failure1.q ql/src/test/queries/clientnegative/truncate_table_failure2.q ql/src/test/queries/clientnegative/truncate_table_failure3.q ql/src/test/queries/clientnegative/truncate_table_failure4.q ql/src/test/queries/clientpositive/truncate_table.q ql/src/test/results/clientnegative/truncate_table_failure1.q.out ql/src/test/results/clientnegative/truncate_table_failure2.q.out ql/src/test/results/clientnegative/truncate_table_failure3.q.out ql/src/test/results/clientnegative/truncate_table_failure4.q.out ql/src/test/results/clientpositive/truncate_table.q.out To: JIRA, navis Cc: njain, mgrover
          Hide
          Phabricator added a comment -

          navis has commented on the revision "HIVE-446 [jira] Implement TRUNCATE".

          INLINE COMMENTS
          ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java:727 Currently, many of hive DDLs are not supporting specifying database name, which is partly fixed on https://issues.apache.org/jira/browse/HIVE-3589. I'm waiting for the issue committed and expecting to use it for other DDLs.
          ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java:755 I agree but can't think of an alternative way. It's same situation for all other DDLs.
          ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java:3912 I admit I've just copied it from Hive.class. Could you do that when updating patch for HIVE-3701? I don't know well of differences among the hadoop versions.
          ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java:249 Agreed.
          ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java:749 done.

          REVISION DETAIL
          https://reviews.facebook.net/D7371

          To: JIRA, navis
          Cc: njain, mgrover

          Show
          Phabricator added a comment - navis has commented on the revision " HIVE-446 [jira] Implement TRUNCATE". INLINE COMMENTS ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java:727 Currently, many of hive DDLs are not supporting specifying database name, which is partly fixed on https://issues.apache.org/jira/browse/HIVE-3589 . I'm waiting for the issue committed and expecting to use it for other DDLs. ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java:755 I agree but can't think of an alternative way. It's same situation for all other DDLs. ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java:3912 I admit I've just copied it from Hive.class. Could you do that when updating patch for HIVE-3701 ? I don't know well of differences among the hadoop versions. ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java:249 Agreed. ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java:749 done. REVISION DETAIL https://reviews.facebook.net/D7371 To: JIRA, navis Cc: njain, mgrover
          Hide
          Phabricator added a comment -

          mgrover has commented on the revision "HIVE-446 [jira] Implement TRUNCATE".

          INLINE COMMENTS
          ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java:3912 -rmr is deprecated. I think the new hip thing to do is "rm -r". Keep in mind though that if hive is being used with really old versions of hadoop, rm -r may not work, so some version matching needs to happen here.

          If we leave it the way it is, then users will see a deprecation warning. I would prefer that not to be the case (btw, that's presently the case with insert overwrite but HIVE-3701 plans to change that)
          ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java:727 Ok, we should add a test that verifies that something like this works:
          hive>use default;
          hive>truncate some_other_db.my_table;
          ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java:249 I am a little confused here. The code seems to indicate that this only works managed, native tables. For example, you can't have a HBase table (using HBase storage handler) being truncated with this command, can you?
          If so, should we have make error messages for those too?
          ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java:749 Check for non-native tables here?
          ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java:3913 Touché! Since we preserve the metadata, we are preserving it for partitions as well. So, it makes sense to create empty directories for partitions that the metastore thinks exist.
          ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java:755 This could be a bottleneck if the number of partitions is large. Can't it? Is there an alternative to adding each of the partitions individually? Can we still not do
          outputs.add(new WriteEntity(table));

          ?

          REVISION DETAIL
          https://reviews.facebook.net/D7371

          To: JIRA, navis
          Cc: njain, mgrover

          Show
          Phabricator added a comment - mgrover has commented on the revision " HIVE-446 [jira] Implement TRUNCATE". INLINE COMMENTS ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java:3912 -rmr is deprecated. I think the new hip thing to do is "rm -r". Keep in mind though that if hive is being used with really old versions of hadoop, rm -r may not work, so some version matching needs to happen here. If we leave it the way it is, then users will see a deprecation warning. I would prefer that not to be the case (btw, that's presently the case with insert overwrite but HIVE-3701 plans to change that) ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java:727 Ok, we should add a test that verifies that something like this works: hive>use default; hive>truncate some_other_db.my_table; ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java:249 I am a little confused here. The code seems to indicate that this only works managed, native tables. For example, you can't have a HBase table (using HBase storage handler) being truncated with this command, can you? If so, should we have make error messages for those too? ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java:749 Check for non-native tables here? ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java:3913 Touché! Since we preserve the metadata, we are preserving it for partitions as well. So, it makes sense to create empty directories for partitions that the metastore thinks exist. ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java:755 This could be a bottleneck if the number of partitions is large. Can't it? Is there an alternative to adding each of the partitions individually? Can we still not do outputs.add(new WriteEntity(table)); ? REVISION DETAIL https://reviews.facebook.net/D7371 To: JIRA, navis Cc: njain, mgrover
          Hide
          Phabricator added a comment -

          navis updated the revision "HIVE-446 [jira] Implement TRUNCATE".
          Reviewers: JIRA

          Addressed comments

          REVISION DETAIL
          https://reviews.facebook.net/D7371

          AFFECTED FILES
          ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
          ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java
          ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g
          ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java
          ql/src/java/org/apache/hadoop/hive/ql/plan/DDLWork.java
          ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java
          ql/src/java/org/apache/hadoop/hive/ql/plan/TruncateTableDesc.java
          ql/src/test/queries/clientnegative/truncate_table_failure1.q
          ql/src/test/queries/clientnegative/truncate_table_failure2.q
          ql/src/test/queries/clientnegative/truncate_table_failure3.q
          ql/src/test/queries/clientpositive/truncate_table.q
          ql/src/test/results/clientnegative/truncate_table_failure1.q.out
          ql/src/test/results/clientnegative/truncate_table_failure2.q.out
          ql/src/test/results/clientnegative/truncate_table_failure3.q.out
          ql/src/test/results/clientpositive/truncate_table.q.out

          To: JIRA, navis
          Cc: njain

          Show
          Phabricator added a comment - navis updated the revision " HIVE-446 [jira] Implement TRUNCATE". Reviewers: JIRA Addressed comments REVISION DETAIL https://reviews.facebook.net/D7371 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java ql/src/java/org/apache/hadoop/hive/ql/plan/DDLWork.java ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java ql/src/java/org/apache/hadoop/hive/ql/plan/TruncateTableDesc.java ql/src/test/queries/clientnegative/truncate_table_failure1.q ql/src/test/queries/clientnegative/truncate_table_failure2.q ql/src/test/queries/clientnegative/truncate_table_failure3.q ql/src/test/queries/clientpositive/truncate_table.q ql/src/test/results/clientnegative/truncate_table_failure1.q.out ql/src/test/results/clientnegative/truncate_table_failure2.q.out ql/src/test/results/clientnegative/truncate_table_failure3.q.out ql/src/test/results/clientpositive/truncate_table.q.out To: JIRA, navis Cc: njain
          Hide
          Phabricator added a comment -

          njain has commented on the revision "HIVE-446 [jira] Implement TRUNCATE".

          INLINE COMMENTS
          ql/src/test/queries/clientpositive/truncate_table.q:3 Isn't readnputs and writeInputs populated correctly for truncate ?

          If yes, shouldn't these commands fail for src/srcpart etc.
          ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java:748 I remember you added a new function getTable() to do all error checking etc.
          Cant you re-use that ?
          ql/src/test/queries/clientpositive/truncate_table.q:8 What is the behavior for a non-partitioned table, when partition spec. is not specified ?
          Either throw an error, or truncate all partitions ?

          Support specifying partial partition specifications:

          truncate table srcpart partition (ds='2008-04-08' hr);
          ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java:759 It should be possible to specify partial partn specifications
          ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java:754 add negative test for this.
          ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java:3895 inputs/outputs not populated

          REVISION DETAIL
          https://reviews.facebook.net/D7371

          To: JIRA, navis
          Cc: njain

          Show
          Phabricator added a comment - njain has commented on the revision " HIVE-446 [jira] Implement TRUNCATE". INLINE COMMENTS ql/src/test/queries/clientpositive/truncate_table.q:3 Isn't readnputs and writeInputs populated correctly for truncate ? If yes, shouldn't these commands fail for src/srcpart etc. ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java:748 I remember you added a new function getTable() to do all error checking etc. Cant you re-use that ? ql/src/test/queries/clientpositive/truncate_table.q:8 What is the behavior for a non-partitioned table, when partition spec. is not specified ? Either throw an error, or truncate all partitions ? Support specifying partial partition specifications: truncate table srcpart partition (ds='2008-04-08' hr); ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java:759 It should be possible to specify partial partn specifications ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java:754 add negative test for this. ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java:3895 inputs/outputs not populated REVISION DETAIL https://reviews.facebook.net/D7371 To: JIRA, navis Cc: njain
          Hide
          Namit Jain added a comment -

          comments on phabricator

          Show
          Namit Jain added a comment - comments on phabricator
          Hide
          Phabricator added a comment -

          navis requested code review of "HIVE-446 [jira] Implement TRUNCATE".
          Reviewers: JIRA

          DPAL-1947 Implement TRUNCATE

          truncate the data but leave the table and metadata intact.

          TEST PLAN
          EMPTY

          REVISION DETAIL
          https://reviews.facebook.net/D7371

          AFFECTED FILES
          ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
          ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java
          ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g
          ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java
          ql/src/java/org/apache/hadoop/hive/ql/plan/DDLWork.java
          ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java
          ql/src/java/org/apache/hadoop/hive/ql/plan/TruncateTableDesc.java
          ql/src/test/queries/clientpositive/truncate_table.q
          ql/src/test/results/clientpositive/truncate_table.q.out

          MANAGE HERALD DIFFERENTIAL RULES
          https://reviews.facebook.net/herald/view/differential/

          WHY DID I GET THIS EMAIL?
          https://reviews.facebook.net/herald/transcript/17631/

          To: JIRA, navis

          Show
          Phabricator added a comment - navis requested code review of " HIVE-446 [jira] Implement TRUNCATE". Reviewers: JIRA DPAL-1947 Implement TRUNCATE truncate the data but leave the table and metadata intact. TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D7371 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java ql/src/java/org/apache/hadoop/hive/ql/plan/DDLWork.java ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java ql/src/java/org/apache/hadoop/hive/ql/plan/TruncateTableDesc.java ql/src/test/queries/clientpositive/truncate_table.q ql/src/test/results/clientpositive/truncate_table.q.out MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/17631/ To: JIRA, navis
          Hide
          Adam Kramer added a comment -

          One year later, I still want this feature.

          Here's a spec:

          TRUNCATE TABLE tablename;
          ...would delete all the data in it (simple hadoop dfs -rmr), OR drop all of the partitions if it is a partitioned table (less simple).

          TRUNCATE TABLE tablename PARTITION(ds='2011-01-01') would delete all the data in the partition.

          Show
          Adam Kramer added a comment - One year later, I still want this feature. Here's a spec: TRUNCATE TABLE tablename; ...would delete all the data in it (simple hadoop dfs -rmr), OR drop all of the partitions if it is a partitioned table (less simple). TRUNCATE TABLE tablename PARTITION(ds='2011-01-01') would delete all the data in the partition.
          Hide
          Adam Kramer added a comment -

          Another note: Sometimes I shut off hive.merge.mapfiles; because the extra mapreduce step really isn't worth it given that I'm effectively creating a temp table that I don't need to use again. The result is a lot of unnecessary inode usage, because I only need those inodes for one more query.

          If I had this option, I would turn off hive.merge.mapfiles;, run my query, run the dependent query, and then truncate the first table to save inodes.

          Show
          Adam Kramer added a comment - Another note: Sometimes I shut off hive.merge.mapfiles; because the extra mapreduce step really isn't worth it given that I'm effectively creating a temp table that I don't need to use again. The result is a lot of unnecessary inode usage, because I only need those inodes for one more query. If I had this option, I would turn off hive.merge.mapfiles;, run my query, run the dependent query, and then truncate the first table to save inodes.
          Hide
          Adam Kramer added a comment -

          This workaround doesn't work for tables with partitions.

          TRUNCATE TABLE tablename;

          should also drop every partition in it for partitioned tables.

          Show
          Adam Kramer added a comment - This workaround doesn't work for tables with partitions. TRUNCATE TABLE tablename; should also drop every partition in it for partitioned tables.
          Hide
          Prasad Chakka added a comment -

          work around from matt prestritto:
          Insert overwrite tbl select * from tbl where col1 = 'fake value';

          Show
          Prasad Chakka added a comment - work around from matt prestritto: Insert overwrite tbl select * from tbl where col1 = 'fake value';

            People

            • Assignee:
              Navis
              Reporter:
              Prasad Chakka
            • Votes:
              5 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development