Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1012

documentLocation attribute in LdapEntry for HDFSProxy isn't specific to a cluster

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.1, 0.20.2, 0.21.0, 0.22.0
    • Fix Version/s: 0.21.0
    • Component/s: contrib/hdfsproxy
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Support for fully qualified HDFS path in addition to simple unqualified path.
      The qualified path indicates that the path is accessible on the specific HDFS. Non qualified path is qualified in all clusters.
      Show
      Support for fully qualified HDFS path in addition to simple unqualified path. The qualified path indicates that the path is accessible on the specific HDFS. Non qualified path is qualified in all clusters.

      Description

      List of allowed document locations accessible through HDFSProxy isn't specific to a cluster. LDAP entries can include the name of the cluster to which the path belongs to have better control on which clusters/paths are accessible through HDFSProxy by the user.

      1. HDFS-1012-bp-y20s.patch
        4 kB
        Srikanth Sundarrajan
      2. HDFS-1012-bp-y20.patch
        4 kB
        Srikanth Sundarrajan
      3. HDFS-1012-bp-y20.patch
        8 kB
        Srikanth Sundarrajan
      4. HDFS-1012.patch
        8 kB
        Srikanth Sundarrajan
      5. HDFS-1012.patch
        4 kB
        Srikanth Sundarrajan
      6. HDFS-1012-bp-y20s.patch
        5 kB
        Srikanth Sundarrajan
      7. HDFS-1012-bp-y20.patch
        4 kB
        Srikanth Sundarrajan

        Issue Links

          Activity

          Hide
          Ramesh Sekaran added a comment -

          The changes for this jira are made upon the changes for HDFS-481, which covers some critical fixes

          Show
          Ramesh Sekaran added a comment - The changes for this jira are made upon the changes for HDFS-481 , which covers some critical fixes
          Hide
          Ramesh Sekaran added a comment -

          Unable to upload patch. Will get the patch uploaded by others

          Show
          Ramesh Sekaran added a comment - Unable to upload patch. Will get the patch uploaded by others
          Hide
          Ramesh Sekaran added a comment -

          Support for fully qualified HDFS path in addition to simple unqualified path.
          The qualified path indicates that the path is accessible on the specific HDFS. Non qualified path is qualified in all clusters.

          Show
          Ramesh Sekaran added a comment - Support for fully qualified HDFS path in addition to simple unqualified path. The qualified path indicates that the path is accessible on the specific HDFS. Non qualified path is qualified in all clusters.
          Hide
          Srikanth Sundarrajan added a comment -

          >> Uploading patch for Ramesh

          It should be possible with this patch to fully qualify the documentLocation attorbute in ldap to include the cluster name (fs.default.name). If the documentLocation in ldap is fully qualified, the access is restricted by the proxy to that specific cluster, If the documentLocation is not qualified, then access is not restricted to a cluster.

          Backport patches: HDFS-1012-bp-y20.patch & HDFS-1012-bp-y20s.patch are not for commit.

          Show
          Srikanth Sundarrajan added a comment - >> Uploading patch for Ramesh It should be possible with this patch to fully qualify the documentLocation attorbute in ldap to include the cluster name (fs.default.name). If the documentLocation in ldap is fully qualified, the access is restricted by the proxy to that specific cluster, If the documentLocation is not qualified, then access is not restricted to a cluster. Backport patches: HDFS-1012 -bp-y20.patch & HDFS-1012 -bp-y20s.patch are not for commit.
          Hide
          Srikanth Sundarrajan added a comment -

          Patches for HDFS-481 & HDFS-1010 need to be applied before this patch

          Show
          Srikanth Sundarrajan added a comment - Patches for HDFS-481 & HDFS-1010 need to be applied before this patch
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12439270/HDFS-1012.patch
          against trunk revision 925004.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/275/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12439270/HDFS-1012.patch against trunk revision 925004. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/275/console This message is automatically generated.
          Hide
          Srikanth Sundarrajan added a comment -

          Note: HDFS-481, HDFS-1009 and HDFS-1010 will need to be committed before this patch can be applied

          Output from test-patch & test-contrib

          [exec] +1 overall.
          [exec]
          [exec] +1 @author. The patch does not contain any @author tags.
          [exec]
          [exec] +1 tests included. The patch appears to include 4 new or modified tests.
          [exec]
          [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
          [exec]
          [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
          [exec]
          [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
          [exec]
          [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.

          test:

          BUILD SUCCESSFUL
          Total time: 4 minutes 32 seconds

          Show
          Srikanth Sundarrajan added a comment - Note: HDFS-481 , HDFS-1009 and HDFS-1010 will need to be committed before this patch can be applied Output from test-patch & test-contrib [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 4 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. test: BUILD SUCCESSFUL Total time: 4 minutes 32 seconds
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Here is my review comments:

          • HDFS_PATH_PATTERN would not work if the port number is omitted.
          • I suggest to store the namenode string in init(..). Then, conf could become a local variable in init(..)
          • The current test seems not covering all of the cases.
          Show
          Tsz Wo Nicholas Sze added a comment - Here is my review comments: HDFS_PATH_PATTERN would not work if the port number is omitted. I suggest to store the namenode string in init(..). Then, conf could become a local variable in init(..) The current test seems not covering all of the cases.
          Hide
          Srikanth Sundarrajan added a comment -
          • HDFS_PATH_PATTERN would not work if the port number is omitted.
          • I suggest to store the namenode string in init(..). Then, conf could become a local variable in init(..)
          • The current test seems not covering all of the cases.

          Nicholas, Thanks for your review comments. Revised patch uploaded with the following changes

          1. HDFS_PATH_PATTERN replaced from ^hdfs://(\\w\\-(\\.)?):
          d+ to (^hdfs://(\\w\\-(\\.)?):\\d+|^hdfs://(\\w\\-(
          .)?)
          ) to support cases with no port #

          2. conf.get("fs.default.name") is retrieved once in init and namenode url stored for future reference and access. This doesn't change within the filter while webapp context is running

          3. Three additional test cases have been added

          • Case for allowing a request with a valid unqualified documentLocation (path)
          • Case for allowing a request with a valid qualified documentLocation
          • Case for rejecting request where documentLocation doesn't contain a valid path for the cluster in question

          ------ Output from test-patch & test-contrib ------

          [exec] +1 overall.
          [exec]
          [exec] +1 @author. The patch does not contain any @author tags.
          [exec]
          [exec] +1 tests included. The patch appears to include 5 new or modified tests.
          [exec]
          [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
          [exec]
          [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
          [exec]
          [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
          [exec]
          [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.

          [cactus] Running org.apache.hadoop.hdfsproxy.TestAuthorizationFilter
          [cactus] Tomcat 5.x started on port [30300]
          [cactus] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 0.972 sec
          [cactus] Running org.apache.hadoop.hdfsproxy.TestLdapIpDirFilter
          [cactus] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.789 sec
          [cactus] Running org.apache.hadoop.hdfsproxy.TestProxyFilter
          [cactus] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.867 sec
          [cactus] Running org.apache.hadoop.hdfsproxy.TestProxyForwardServlet
          [cactus] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.508 sec
          [cactus] Running org.apache.hadoop.hdfsproxy.TestProxyUtil
          [cactus] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 1.84 sec
          [cactus] Tomcat 5.x is stopping...
          [cactus] Tomcat 5.x is stopped

          test:

          BUILD SUCCESSFUL
          Total time: 4 minutes 8 seconds

          Show
          Srikanth Sundarrajan added a comment - HDFS_PATH_PATTERN would not work if the port number is omitted. I suggest to store the namenode string in init(..). Then, conf could become a local variable in init(..) The current test seems not covering all of the cases. Nicholas, Thanks for your review comments. Revised patch uploaded with the following changes 1. HDFS_PATH_PATTERN replaced from ^hdfs://( \\w\\- (\\.)?) : d+ to (^hdfs://( \\w\\- (\\.)?) :\\d+|^hdfs://( \\w\\- ( .)?) ) to support cases with no port # 2. conf.get("fs.default.name") is retrieved once in init and namenode url stored for future reference and access. This doesn't change within the filter while webapp context is running 3. Three additional test cases have been added Case for allowing a request with a valid unqualified documentLocation (path) Case for allowing a request with a valid qualified documentLocation Case for rejecting request where documentLocation doesn't contain a valid path for the cluster in question ------ Output from test-patch & test-contrib ------ [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 5 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [cactus] Running org.apache.hadoop.hdfsproxy.TestAuthorizationFilter [cactus] Tomcat 5.x started on port [30300] [cactus] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 0.972 sec [cactus] Running org.apache.hadoop.hdfsproxy.TestLdapIpDirFilter [cactus] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.789 sec [cactus] Running org.apache.hadoop.hdfsproxy.TestProxyFilter [cactus] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.867 sec [cactus] Running org.apache.hadoop.hdfsproxy.TestProxyForwardServlet [cactus] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.508 sec [cactus] Running org.apache.hadoop.hdfsproxy.TestProxyUtil [cactus] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 1.84 sec [cactus] Tomcat 5.x is stopping... [cactus] Tomcat 5.x is stopped test: BUILD SUCCESSFUL Total time: 4 minutes 8 seconds
          Hide
          Tsz Wo Nicholas Sze added a comment -

          +1 the new patch is perfect!

          Show
          Tsz Wo Nicholas Sze added a comment - +1 the new patch is perfect!
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I have committed this. Thanks, Srikanth!

          Show
          Tsz Wo Nicholas Sze added a comment - I have committed this. Thanks, Srikanth!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #234 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/234/)
          . hdfsproxy: Support for fully qualified HDFS path in addition to simple unqualified path. Contributed by Srikanth Sundarrajan

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #234 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/234/ ) . hdfsproxy: Support for fully qualified HDFS path in addition to simple unqualified path. Contributed by Srikanth Sundarrajan
          Hide
          Srikanth Sundarrajan added a comment -

          Updated backport patch with the review comments

          Show
          Srikanth Sundarrajan added a comment - Updated backport patch with the review comments
          Hide
          Srikanth Sundarrajan added a comment -

          Revised backport patch for yhadoop20

          Show
          Srikanth Sundarrajan added a comment - Revised backport patch for yhadoop20
          Hide
          Srikanth Sundarrajan added a comment -

          Revised backport patch for yhadoop20s in sync with trunk patch revisions

          Show
          Srikanth Sundarrajan added a comment - Revised backport patch for yhadoop20s in sync with trunk patch revisions

            People

            • Assignee:
              Srikanth Sundarrajan
              Reporter:
              Srikanth Sundarrajan
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development