Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-447

proxy to call LDAP for IP lookup and get user ID and directories, validate requested URL

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: contrib/hdfsproxy
    • Labels:
      None

      Description

      It is easy to manage user accounts using LDAP. by adding support for LDAP, proxy can do IP authorization in a headless fashion.

      when a user send a request, proxy extract IP address and request PathInfo from the request. then it searches the LDAP server to get the allowed HDFS root paths given the IP address. Proxy will match the user request PathInfo with the allowed HDFS root path, return 403 if it could not find a match.

      1. HDFS-447.patch
        71 kB
        zhiyong zhang
      2. HDFS-447.patch
        70 kB
        zhiyong zhang
      3. HADOOP-5851.patch
        70 kB
        zhiyong zhang
      4. HADOOP-5851.patch
        70 kB
        zhiyong zhang
      5. HADOOP-5851.patch
        70 kB
        zhiyong zhang
      6. HADOOP-5851.patch
        68 kB
        zhiyong zhang
      7. HADOOP-5851.patch
        60 kB
        zhiyong zhang
      8. HADOOP-5851.patch
        39 kB
        zhiyong zhang
      9. HADOOP-5851.patch
        39 kB
        zhiyong zhang

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #8 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/8/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #8 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/8/ )
          Hide
          Chris Douglas added a comment -

          +1

          test-patch and all hdfsproxy unit tests passed. The changes outside the contrib package are exercised only by the aforementioned tests.

          I committed this. Thanks Zhiyong!

          Show
          Chris Douglas added a comment - +1 test-patch and all hdfsproxy unit tests passed. The changes outside the contrib package are exercised only by the aforementioned tests. I committed this. Thanks Zhiyong!
          Hide
          zhiyong zhang added a comment -

          in case there is no "ssl.client.keystore.location" defined, use empty keymanager instead of throws exception when reading keystore file.
          will avoid exception in case server does not need to authenticate client.

          Show
          zhiyong zhang added a comment - in case there is no "ssl.client.keystore.location" defined, use empty keymanager instead of throws exception when reading keystore file. will avoid exception in case server does not need to authenticate client.
          Hide
          zhiyong zhang added a comment -

          I just run the test.
          -1 overall.
          [exec]
          [exec] +1 @author. The patch does not contain any @author tags.
          [exec]
          [exec] +1 tests included. The patch appears to include 27 new or modified tests.
          [exec]
          [exec] -1 javadoc. The javadoc tool appears to have generated 1 warning messages.
          [exec]
          [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
          [exec]
          [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
          [exec]
          [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.

          the javadoc warning is from src/java/org/apache/hadoop/hdfs/DFSClient.java:396: warning - Tag @link: can't find create(String,FsPermission,boolean,short,long,Progressable,int) in org.apache.hadoop.hdfs.DFSClient
          and src/java/org/apache/hadoop/hdfs/DFSClient.java:422: warning - Tag @see: can't find create(String, FsPermission, String, boolean, short, long) in org.apache.hadoop.hdfs.protocol.ClientProtocol

          not from the patch.

          Show
          zhiyong zhang added a comment - I just run the test. -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 27 new or modified tests. [exec] [exec] -1 javadoc. The javadoc tool appears to have generated 1 warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. the javadoc warning is from src/java/org/apache/hadoop/hdfs/DFSClient.java:396: warning - Tag @link: can't find create(String,FsPermission,boolean,short,long,Progressable,int) in org.apache.hadoop.hdfs.DFSClient and src/java/org/apache/hadoop/hdfs/DFSClient.java:422: warning - Tag @see: can't find create(String, FsPermission, String, boolean, short, long) in org.apache.hadoop.hdfs.protocol.ClientProtocol not from the patch.
          Hide
          zhiyong zhang added a comment -

          remove find-bug warnings.

          Show
          zhiyong zhang added a comment - remove find-bug warnings.
          Hide
          zhiyong zhang added a comment -

          corrected several typos and changed several places for code optimization.

          Show
          zhiyong zhang added a comment - corrected several typos and changed several places for code optimization.
          Hide
          zhiyong zhang added a comment -

          1. removed System.setProperty. Use Keymanagers and TrustManagers instead.

          2. re-patched in new hdfs trunk after project split.

          3. replaced the printStackTrace with normal LOG.debug(e.toString()).

          Show
          zhiyong zhang added a comment - 1. removed System.setProperty. Use Keymanagers and TrustManagers instead. 2. re-patched in new hdfs trunk after project split. 3. replaced the printStackTrace with normal LOG.debug(e.toString()).
          Hide
          Philip Zeyliger added a comment -

          I'm traveling overseas and will return on June 29th. For urgent
          matters, please contact Amr Awadallah.

          Show
          Philip Zeyliger added a comment - I'm traveling overseas and will return on June 29th. For urgent matters, please contact Amr Awadallah.
          Hide
          Chris Douglas added a comment -

          The patch needs to be regenerated for the code split.

          The contents look fine, though I'd like to see the TrustManager path completely replace the System.setProperty idiom now that it is understood. The only other minor nit would be here:

          +        if (LOG.isDebugEnabled())
          +          e.printStackTrace();
          

          Using LOG.debug("Useful message", e) is more standard.

          Show
          Chris Douglas added a comment - The patch needs to be regenerated for the code split. The contents look fine, though I'd like to see the TrustManager path completely replace the System.setProperty idiom now that it is understood. The only other minor nit would be here: + if (LOG.isDebugEnabled()) + e.printStackTrace(); Using LOG.debug("Useful message", e) is more standard.
          Hide
          zhiyong zhang added a comment -

          1.) As suggested by Kan, trust all server certs should not be the default setting.

          Add
          <property>
          <name>ssl.client.do.not.authenticate.server</name>
          <value>false</value>
          <description> if true, trust all server certificates
          </description>
          </property>
          in ssl-client.xml configuration. By default, still need to validate server certificates.

          2.) Did a code walk-through with Rob. W. Also discussed with Kan, the should get the group information from LDAP server instead of using proxyUgiManager. changed this part so that proxy will pass HDFS userId and group info to source cluster.

          3.) Merged all configuration files into one configuration file (hdfsproxy-default.xml), this saves some effort to manage the configuration files and war files.
          The configuration in hdfsproxy-default.xml should contain
          <name>fs.default.name</name>
          <name>dfs.block.size</name>
          <name>io.file.buffer.size</name>
          in addition to ldap-based properties.

          Show
          zhiyong zhang added a comment - 1.) As suggested by Kan, trust all server certs should not be the default setting. Add <property> <name>ssl.client.do.not.authenticate.server</name> <value>false</value> <description> if true, trust all server certificates </description> </property> in ssl-client.xml configuration. By default, still need to validate server certificates. 2.) Did a code walk-through with Rob. W. Also discussed with Kan, the should get the group information from LDAP server instead of using proxyUgiManager. changed this part so that proxy will pass HDFS userId and group info to source cluster. 3.) Merged all configuration files into one configuration file (hdfsproxy-default.xml), this saves some effort to manage the configuration files and war files. The configuration in hdfsproxy-default.xml should contain <name>fs.default.name</name> <name>dfs.block.size</name> <name>io.file.buffer.size</name> in addition to ldap-based properties.
          Hide
          Kan Zhang added a comment -

          I can see that for encryption purpose you may still want to use SSL. In that case, I suggest you make it an option instead of the default behavior. The user has to explicitly specify this option to enable it.

          Show
          Kan Zhang added a comment - I can see that for encryption purpose you may still want to use SSL. In that case, I suggest you make it an option instead of the default behavior. The user has to explicitly specify this option to enable it.
          Hide
          Kan Zhang added a comment -

          @zhiyong, why you want to do that? If SSL authentication is not needed, one can use HFTP instead of HSFTP.

          Show
          Kan Zhang added a comment - @zhiyong, why you want to do that? If SSL authentication is not needed, one can use HFTP instead of HSFTP.
          Hide
          zhiyong zhang added a comment -

          in client side (HsftpFileSystem.java), use a DummyTrustStoreManager to accept all server certificates, in case
          ssl-client.xml is not defined.

          like curl's -k option.

          use this client, user will not need to worry about importing server CA to truststore.

          Show
          zhiyong zhang added a comment - in client side (HsftpFileSystem.java), use a DummyTrustStoreManager to accept all server certificates, in case ssl-client.xml is not defined. like curl's -k option. use this client, user will not need to worry about importing server CA to truststore.
          Hide
          zhiyong zhang added a comment -

          use hostname.split("[-\\.]") to extract servlet context path (war file name)

          Show
          zhiyong zhang added a comment - use hostname.split(" [-\\.] ") to extract servlet context path (war file name)
          Hide
          zhiyong zhang added a comment -

          a) add another filter code named LdapIpDirFilter.java, will be installed as a deployment descriptor (in web.xml).

          b) program flowchart.
          1. IP = HttpServletRequest.getRemoteAddr()
          2. Ldap.search(uniqueMember:cn=IP) to find role.
          2.a. if nothing found, 403 return
          3. role A was found, from role A, extract userId and HDFS root paths allowed, say PLIST.
          4. log userId.
          5. get PATH = HttpServletRequest.getPathInfo()
          6. check if PATH or its parents is in PLIST
          6.a. if not, 403 return.
          7. access allowed, filter pass

          c) use a Dummy LDAP Object server to do the unit test.

          Show
          zhiyong zhang added a comment - a) add another filter code named LdapIpDirFilter.java, will be installed as a deployment descriptor (in web.xml). b) program flowchart. 1. IP = HttpServletRequest.getRemoteAddr() 2. Ldap.search(uniqueMember:cn=IP) to find role. 2.a. if nothing found, 403 return 3. role A was found, from role A, extract userId and HDFS root paths allowed, say PLIST. 4. log userId. 5. get PATH = HttpServletRequest.getPathInfo() 6. check if PATH or its parents is in PLIST 6.a. if not, 403 return. 7. access allowed, filter pass c) use a Dummy LDAP Object server to do the unit test.

            People

            • Assignee:
              zhiyong zhang
              Reporter:
              zhiyong zhang
            • Votes:
              1 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Due:
                Created:
                Updated:
                Resolved:

                Development