Hive
  1. Hive
  2. HIVE-1444

"hdfs" is hardcoded in few places in the code which inhibits use of other file systems

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.3.0, 0.4.0, 0.4.1, 0.5.0, 0.6.0, 0.7.0
    • Fix Version/s: 0.9.0
    • Component/s: Query Processor
    • Labels:
      None
    • Environment:

      any

      Description

      In quite a few places "hdfs" is hardcoded, which is OK for majority of the cases, except when it is not really hdfs, but s3 or any other file system.

      The place where it really breaks is:
      in ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java :

      method: private void applyConstraints(URI fromURI, URI toURI, Tree ast, boolean isLocal)

      First few lines are check for file system:
      if (!fromURI.getScheme().equals("file")
      && !fromURI.getScheme().equals("hdfs"))

      { throw new SemanticException(ErrorMsg.INVALID_PATH.getMsg(ast, "only \"file\" or \"hdfs\" file systems accepted")); }

      "hdfs" is hardcoded.

      I don't think you need to have this check at all as you are checking whether filesystem is local or not later on anyway and in regards to non locla file system - if one would be bad one you would get problems or have it look like local before you even come to "applyConstraints" method.

        Activity

        Hide
        T Jake Luciani added a comment -

        bump

        Show
        T Jake Luciani added a comment - bump
        Hide
        Phabricator added a comment -

        edwardcapriolo requested code review of "HIVE-1444 [jira] "hdfs" is hardcoded in few places in the code which inhibits use of other file systems".
        Reviewers: JIRA

        https://issues.apache.org/jira/browse/HIVE-1444

        Allows other fs implementations then HDFS

        In quite a few places "hdfs" is hardcoded, which is OK for majority of the cases, except when it is not really hdfs, but s3 or any other file system.

        The place where it really breaks is:
        in ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java :

        method: private void applyConstraints(URI fromURI, URI toURI, Tree ast, boolean isLocal)

        First few lines are check for file system:
        if (!fromURI.getScheme().equals("file")
        && !fromURI.getScheme().equals("hdfs"))

        { throw new SemanticException(ErrorMsg.INVALID_PATH.getMsg(ast, "only \"file\" or \"hdfs\" file systems accepted")); }

        "hdfs" is hardcoded.

        I don't think you need to have this check at all as you are checking whether filesystem is local or not later on anyway and in regards to non locla file system - if one would be bad one you would get problems or have it look like local before you even come to "applyConstraints" method.

        TEST PLAN
        EMPTY

        REVISION DETAIL
        https://reviews.facebook.net/D1839

        AFFECTED FILES
        ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java

        MANAGE HERALD DIFFERENTIAL RULES
        https://reviews.facebook.net/herald/view/differential/

        WHY DID I GET THIS EMAIL?
        https://reviews.facebook.net/herald/transcript/3909/

        Tip: use the X-Herald-Rules header to filter Herald messages in your client.

        Show
        Phabricator added a comment - edwardcapriolo requested code review of " HIVE-1444 [jira] "hdfs" is hardcoded in few places in the code which inhibits use of other file systems". Reviewers: JIRA https://issues.apache.org/jira/browse/HIVE-1444 Allows other fs implementations then HDFS In quite a few places "hdfs" is hardcoded, which is OK for majority of the cases, except when it is not really hdfs, but s3 or any other file system. The place where it really breaks is: in ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java : method: private void applyConstraints(URI fromURI, URI toURI, Tree ast, boolean isLocal) First few lines are check for file system: if (!fromURI.getScheme().equals("file") && !fromURI.getScheme().equals("hdfs")) { throw new SemanticException(ErrorMsg.INVALID_PATH.getMsg(ast, "only \"file\" or \"hdfs\" file systems accepted")); } "hdfs" is hardcoded. I don't think you need to have this check at all as you are checking whether filesystem is local or not later on anyway and in regards to non locla file system - if one would be bad one you would get problems or have it look like local before you even come to "applyConstraints" method. TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D1839 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/3909/ Tip: use the X-Herald-Rules header to filter Herald messages in your client.
        Hide
        Phabricator added a comment -

        ashutoshc has accepted the revision "HIVE-1444 [jira] "hdfs" is hardcoded in few places in the code which inhibits use of other file systems".

        +1 Will commit if tests pass.

        REVISION DETAIL
        https://reviews.facebook.net/D1839

        BRANCH
        svn

        Show
        Phabricator added a comment - ashutoshc has accepted the revision " HIVE-1444 [jira] "hdfs" is hardcoded in few places in the code which inhibits use of other file systems". +1 Will commit if tests pass. REVISION DETAIL https://reviews.facebook.net/D1839 BRANCH svn
        Hide
        Ashutosh Chauhan added a comment -

        Committed to trunk. Thanks, Ed!

        Show
        Ashutosh Chauhan added a comment - Committed to trunk. Thanks, Ed!
        Hide
        Ashutosh Chauhan added a comment -

        This issue is closed now. It was released with the fix in 0.9.0. If there is a problem, please open a new jira and link this one with that.

        Show
        Ashutosh Chauhan added a comment - This issue is closed now. It was released with the fix in 0.9.0. If there is a problem, please open a new jira and link this one with that.
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/)
        HIVE-1444: hdfs is hardcoded in few places in the code which inhibits use of other file systems (ecapriolo via hashutosh) (Revision 1297808)

        Result = ABORTED
        hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297808
        Files :

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java
        Show
        Hudson added a comment - Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/ ) HIVE-1444 : hdfs is hardcoded in few places in the code which inhibits use of other file systems (ecapriolo via hashutosh) (Revision 1297808) Result = ABORTED hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297808 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java

          People

          • Assignee:
            Edward Capriolo
            Reporter:
            Yuliya Feldman
          • Votes:
            1 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development