Hive
  1. Hive
  2. HIVE-2837

insert into external tables should not be allowed

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.9.0
    • Component/s: None
    • Labels:
      None

      Description

      This is a very risky thing to allow.
      Since, the external tables can point to any user location, which can potentially corrupt some other tables.

        Activity

        Hide
        Edward Capriolo added a comment -

        maybe add a read only keyword or table property?

        Show
        Edward Capriolo added a comment - maybe add a read only keyword or table property?
        Hide
        Namit Jain added a comment -

        I dont want to it table-by-table.
        The only other option I can think of is add another configuration variable.

        Show
        Namit Jain added a comment - I dont want to it table-by-table. The only other option I can think of is add another configuration variable.
        Hide
        Edward Capriolo added a comment -

        Internal tables can have a location as well.

        Show
        Edward Capriolo added a comment - Internal tables can have a location as well.
        Hide
        Alan Gates added a comment -

        If this is to be added it should definitely be configurable, with the default off, since this is a major backwards incompatibility.

        Show
        Alan Gates added a comment - If this is to be added it should definitely be configurable, with the default off, since this is a major backwards incompatibility.
        Hide
        Phabricator added a comment -

        njain requested code review of "HIVE-2837 [jira] insert into external tables should not be allowed".
        Reviewers: JIRA

        https://issues.apache.org/jira/browse/HIVE-2837

        HIVE-2837 By default, the behavior does not change. It is backward compatible

        This is a very risky thing to allow.
        Since, the external tables can point to any user location, which can potentially corrupt some other tables.

        TEST PLAN
        EMPTY

        REVISION DETAIL
        https://reviews.facebook.net/D2211

        AFFECTED FILES
        common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
        ql/src/test/results/clientnegative/insertexternal1.q.out
        ql/src/test/queries/clientnegative/insertexternal1.q
        ql/src/java/org/apache/hadoop/hive/ql/parse/ErrorMsg.java
        ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java

        MANAGE HERALD DIFFERENTIAL RULES
        https://reviews.facebook.net/herald/view/differential/

        WHY DID I GET THIS EMAIL?
        https://reviews.facebook.net/herald/transcript/4845/

        Tip: use the X-Herald-Rules header to filter Herald messages in your client.

        Show
        Phabricator added a comment - njain requested code review of " HIVE-2837 [jira] insert into external tables should not be allowed". Reviewers: JIRA https://issues.apache.org/jira/browse/HIVE-2837 HIVE-2837 By default, the behavior does not change. It is backward compatible This is a very risky thing to allow. Since, the external tables can point to any user location, which can potentially corrupt some other tables. TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D2211 AFFECTED FILES common/src/java/org/apache/hadoop/hive/conf/HiveConf.java ql/src/test/results/clientnegative/insertexternal1.q.out ql/src/test/queries/clientnegative/insertexternal1.q ql/src/java/org/apache/hadoop/hive/ql/parse/ErrorMsg.java ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/4845/ Tip: use the X-Herald-Rules header to filter Herald messages in your client.
        Hide
        Namit Jain added a comment -

        I have added a new parameter: hive.insert.into.external.tables which is true by default.
        If you set it to false explicitly, the inserts will fail.

        Show
        Namit Jain added a comment - I have added a new parameter: hive.insert.into.external.tables which is true by default. If you set it to false explicitly, the inserts will fail.
        Hide
        Phabricator added a comment -

        kevinwilfong has added CCs to the revision "HIVE-2837 [jira] insert into external tables should not be allowed".
        Added CCs: kevinwilfong

        REVISION DETAIL
        https://reviews.facebook.net/D2211

        Show
        Phabricator added a comment - kevinwilfong has added CCs to the revision " HIVE-2837 [jira] insert into external tables should not be allowed". Added CCs: kevinwilfong REVISION DETAIL https://reviews.facebook.net/D2211
        Hide
        Phabricator added a comment -

        njain updated the revision "HIVE-2837 [jira] insert into external tables should not be allowed".
        Reviewers: JIRA

        Change error message

        REVISION DETAIL
        https://reviews.facebook.net/D2211

        AFFECTED FILES
        common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
        ql/src/test/results/clientnegative/insertexternal1.q.out
        ql/src/test/queries/clientnegative/insertexternal1.q
        ql/src/java/org/apache/hadoop/hive/ql/parse/ErrorMsg.java
        ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java

        Show
        Phabricator added a comment - njain updated the revision " HIVE-2837 [jira] insert into external tables should not be allowed". Reviewers: JIRA Change error message REVISION DETAIL https://reviews.facebook.net/D2211 AFFECTED FILES common/src/java/org/apache/hadoop/hive/conf/HiveConf.java ql/src/test/results/clientnegative/insertexternal1.q.out ql/src/test/queries/clientnegative/insertexternal1.q ql/src/java/org/apache/hadoop/hive/ql/parse/ErrorMsg.java ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
        Hide
        Phabricator added a comment -

        kevinwilfong has accepted the revision "HIVE-2837 [jira] insert into external tables should not be allowed".

        +1 Will commit if tests pass.

        REVISION DETAIL
        https://reviews.facebook.net/D2211

        BRANCH
        svn

        Show
        Phabricator added a comment - kevinwilfong has accepted the revision " HIVE-2837 [jira] insert into external tables should not be allowed". +1 Will commit if tests pass. REVISION DETAIL https://reviews.facebook.net/D2211 BRANCH svn
        Hide
        Kevin Wilfong added a comment -

        Committed, thanks Namit.

        Show
        Kevin Wilfong added a comment - Committed, thanks Namit.
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-h0.21 #1302 (See https://builds.apache.org/job/Hive-trunk-h0.21/1302/)
        HIVE-2837. insert into external tables should not be allowed. (namit via kevinwilfong) (Revision 1298936)

        Result = SUCCESS
        kevinwilfong : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1298936
        Files :

        • /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ErrorMsg.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
        • /hive/trunk/ql/src/test/queries/clientnegative/insertexternal1.q
        • /hive/trunk/ql/src/test/results/clientnegative/insertexternal1.q.out
        Show
        Hudson added a comment - Integrated in Hive-trunk-h0.21 #1302 (See https://builds.apache.org/job/Hive-trunk-h0.21/1302/ ) HIVE-2837 . insert into external tables should not be allowed. (namit via kevinwilfong) (Revision 1298936) Result = SUCCESS kevinwilfong : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1298936 Files : /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ErrorMsg.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java /hive/trunk/ql/src/test/queries/clientnegative/insertexternal1.q /hive/trunk/ql/src/test/results/clientnegative/insertexternal1.q.out
        Hide
        Ashutosh Chauhan added a comment -

        This issue is closed now. It was released with the fix in 0.9.0. If there is a problem, please open a new jira and link this one with that.

        Show
        Ashutosh Chauhan added a comment - This issue is closed now. It was released with the fix in 0.9.0. If there is a problem, please open a new jira and link this one with that.
        Hide
        Phabricator added a comment -

        njain has abandoned the revision "HIVE-2837 [jira] insert into external tables should not be allowed".

        REVISION DETAIL
        https://reviews.facebook.net/D2211

        Show
        Phabricator added a comment - njain has abandoned the revision " HIVE-2837 [jira] insert into external tables should not be allowed". REVISION DETAIL https://reviews.facebook.net/D2211
        Hide
        Mark Grover added a comment -

        From looking at this, it seems like it applies to external tables on HDFS as well. Can you please confirm this is correct?
        Technically, insert into should be ok for external tables on HDFS (and S3) as well. Seems like a storage file system level thing to specify whether insert into is applied and implement it.

        Some context for you:
        Historically, there hasn't been any real difference between creating an external table on HDFS vs creating a managed one. However, if we disallow insert into on external tables, that would mean that folks with external tables on HDFS wouldn't be able to make use of insert into functionality even though they should be able to. This would make new tables to be biased towards managed tables but what about the people who already have their entire Hive warehouses based on external tables?

        Show
        Mark Grover added a comment - From looking at this, it seems like it applies to external tables on HDFS as well. Can you please confirm this is correct? Technically, insert into should be ok for external tables on HDFS (and S3) as well. Seems like a storage file system level thing to specify whether insert into is applied and implement it. Some context for you: Historically, there hasn't been any real difference between creating an external table on HDFS vs creating a managed one. However, if we disallow insert into on external tables, that would mean that folks with external tables on HDFS wouldn't be able to make use of insert into functionality even though they should be able to. This would make new tables to be biased towards managed tables but what about the people who already have their entire Hive warehouses based on external tables?
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/)
        HIVE-2837. insert into external tables should not be allowed. (namit via kevinwilfong) (Revision 1298936)

        Result = ABORTED
        kevinwilfong : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1298936
        Files :

        • /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ErrorMsg.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
        • /hive/trunk/ql/src/test/queries/clientnegative/insertexternal1.q
        • /hive/trunk/ql/src/test/results/clientnegative/insertexternal1.q.out
        Show
        Hudson added a comment - Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/ ) HIVE-2837 . insert into external tables should not be allowed. (namit via kevinwilfong) (Revision 1298936) Result = ABORTED kevinwilfong : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1298936 Files : /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ErrorMsg.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java /hive/trunk/ql/src/test/queries/clientnegative/insertexternal1.q /hive/trunk/ql/src/test/results/clientnegative/insertexternal1.q.out

          People

          • Assignee:
            Namit Jain
            Reporter:
            Namit Jain
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development