Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-991

Allow browsing the filesystem over http using delegation tokens

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Assuming the user authenticates to the NameNode in the browser, allow them to browse the file system by adding a delegation token the the url when it is redirected to a datanode.

      1. h-991.patch
        48 kB
        Owen O'Malley
      2. h-991.patch
        48 kB
        Owen O'Malley
      3. h-991.patch
        48 kB
        Owen O'Malley
      4. h-991.patch
        47 kB
        Owen O'Malley
      5. h-991.patch
        29 kB
        Owen O'Malley

        Issue Links

          Activity

          Hide
          Devaraj Das added a comment -

          This is fixed in this patch - https://issues.apache.org/jira/secure/attachment/12440931/HDFS-1007-BP20-fix-3.patch

          But the patch hasn't been forward ported to trunk yet.

          Show
          Devaraj Das added a comment - This is fixed in this patch - https://issues.apache.org/jira/secure/attachment/12440931/HDFS-1007-BP20-fix-3.patch But the patch hasn't been forward ported to trunk yet.
          Hide
          Dmytro Molkov added a comment -

          This patch broke the HftpFileSystem, since in StreamFile.java you are now using name.conf attribute to get the configuration, but since it runs in the DataNode it uses datanode.conf name for the configuration attribute.
          This calls for a HftpFileSystem unittest, since we do not have one this slipped through.

          Show
          Dmytro Molkov added a comment - This patch broke the HftpFileSystem, since in StreamFile.java you are now using name.conf attribute to get the configuration, but since it runs in the DataNode it uses datanode.conf name for the configuration attribute. This calls for a HftpFileSystem unittest, since we do not have one this slipped through.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #275 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/275/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #275 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/275/ )
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #302 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/302/)

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #302 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/302/ )
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #146 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/146/)

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #146 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/146/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #205 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/205/)
          . Allow authentication to the web ui via a delegation token.
          (omalley)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #205 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/205/ ) . Allow authentication to the web ui via a delegation token. (omalley)
          Hide
          Owen O'Malley added a comment -

          I just committed this.

          Show
          Owen O'Malley added a comment - I just committed this.
          Hide
          Devaraj Das added a comment -

          Although i should mention that once HADOOP-6580 is committed, the calls to getDelegationToken will fail unless the ugi has the appropriate authentication method set on it.

          Show
          Devaraj Das added a comment - Although i should mention that once HADOOP-6580 is committed, the calls to getDelegationToken will fail unless the ugi has the appropriate authentication method set on it.
          Hide
          Devaraj Das added a comment -

          Looks good. +1

          Show
          Devaraj Das added a comment - Looks good. +1
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12437190/h-991.patch
          against trunk revision 916534.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/250/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/250/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/250/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/250/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12437190/h-991.patch against trunk revision 916534. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/250/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/250/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/250/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/250/console This message is automatically generated.
          Hide
          Owen O'Malley added a comment -

          Modified the findbugs exclude file to block all of the XSS and HRS, since we handle those with input quoting.

          Show
          Owen O'Malley added a comment - Modified the findbugs exclude file to block all of the XSS and HRS, since we handle those with input quoting.
          Hide
          Owen O'Malley added a comment -

          The findbugs warning are spurious, because we have an input filter that quotes all of the HTTP parameters automatically. We should disable them from the build.

          The contrib failure was cactus' failure to download Tomcat.

          Show
          Owen O'Malley added a comment - The findbugs warning are spurious, because we have an input filter that quotes all of the HTTP parameters automatically. We should disable them from the build. The contrib failure was cactus' failure to download Tomcat.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12437124/h-991.patch
          against trunk revision 916534.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 4 new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/248/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/248/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/248/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/248/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12437124/h-991.patch against trunk revision 916534. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 4 new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/248/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/248/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/248/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/248/console This message is automatically generated.
          Hide
          Owen O'Malley added a comment -

          Check patch through Hudson one more time.

          – Owen

          Show
          Owen O'Malley added a comment - Check patch through Hudson one more time. – Owen
          Hide
          Owen O'Malley added a comment -

          Fix the servlets to use getServletContext rather than the request to call getParameter on.

          Show
          Owen O'Malley added a comment - Fix the servlets to use getServletContext rather than the request to call getParameter on.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12437058/h-991.patch
          against trunk revision 916292.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 4 new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/247/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/247/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/247/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/247/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12437058/h-991.patch against trunk revision 916292. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 4 new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/247/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/247/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/247/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/247/console This message is automatically generated.
          Hide
          Owen O'Malley added a comment -

          This time with --no-prefix.

          Show
          Owen O'Malley added a comment - This time with --no-prefix.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12437052/h-991.patch
          against trunk revision 916292.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 patch. The patch command could not apply the patch.

          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/246/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12437052/h-991.patch against trunk revision 916292. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/246/console This message is automatically generated.
          Hide
          Owen O'Malley added a comment -

          An update to the patch. Under manual testing this works for me. We don't currently have the infrastructure to test code that only runs with security turned on. We need to figure out a way to mock out Kerberos...

          Show
          Owen O'Malley added a comment - An update to the patch. Under manual testing this works for me. We don't currently have the infrastructure to test code that only runs with security turned on. We need to figure out a way to mock out Kerberos...
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12436960/h-991.patch
          against trunk revision 916072.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The patch appears to cause tar ant target to fail.

          -1 findbugs. The patch appears to cause Findbugs to fail.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/245/testReport/
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/245/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/245/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12436960/h-991.patch against trunk revision 916072. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The patch appears to cause tar ant target to fail. -1 findbugs. The patch appears to cause Findbugs to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/245/testReport/ Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/245/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/245/console This message is automatically generated.
          Hide
          Owen O'Malley added a comment -

          Updated to unify the user stuff coming in from the web.

          Impossible to write unit tests for most of this code because they depend on security being turned on, which doesn't work in the unit tests (you'd need kerberos installed locally.)

          Show
          Owen O'Malley added a comment - Updated to unify the user stuff coming in from the web. Impossible to write unit tests for most of this code because they depend on security being turned on, which doesn't work in the unit tests (you'd need kerberos installed locally.)
          Hide
          Owen O'Malley added a comment -

          URLEncoder.encode("/", "UTF-8")

          Yeah, I laughed when I saw that code. I probably should have fixed it.

          URL Creation

          Patches accepted, if you have some framework burning a hole in your pocket.

          Tests

          It is actually really hard to write tests for the servlets.

          Show
          Owen O'Malley added a comment - URLEncoder.encode("/", "UTF-8") Yeah, I laughed when I saw that code. I probably should have fixed it. URL Creation Patches accepted, if you have some framework burning a hole in your pocket. Tests It is actually really hard to write tests for the servlets.
          Hide
          Owen O'Malley added a comment -

          When does the delegation token that the namenode provides (redirectToRandomDataNode()) expire?

          Delegation tokens default to living for 1 day.

          A lot of websites pass security tokens via cookies, because GET parameters tend to get written down in referrer fields and such.

          Jetty doesn't do such logging. smile But as you point out, I don't think we really have a workable choice since we have no guarantee that the datanode can get the cookies set by the namenode. sigh

          Show
          Owen O'Malley added a comment - When does the delegation token that the namenode provides (redirectToRandomDataNode()) expire? Delegation tokens default to living for 1 day. A lot of websites pass security tokens via cookies, because GET parameters tend to get written down in referrer fields and such. Jetty doesn't do such logging. smile But as you point out, I don't think we really have a workable choice since we have no guarantee that the datanode can get the cookies set by the namenode. sigh
          Hide
          Philip Zeyliger added a comment -

          Hi Owen,

          When does the delegation token that the namenode provides (redirectToRandomDataNode()) expire?

          A lot of websites pass security tokens via cookies, because GET parameters tend to get written down in referrer fields and such. So there's the potential that someone will get their hands on your token. Am I right that the token lets anyone read any data as if they were you? I'd be more comfortable if it were cookie based (though that implies that your datanodes and your namenode are in the same domain, which might not be workable), though I do see how GET is simpler.

          The web security part of me is also worried that this is liable to CSRF (http://en.wikipedia.org/wiki/Cross-site_request_forgery) attacks. The key there, I think, is to make sure that when the namenode is issuing tokens, it's absolutely confident that it's issuing them to someone who is asking for them.

          URLEncoder.encode("/", "UTF-8")

          Might be worthwhile to make this a constant. It's unlikely to change

          URL Creation

          This isn't necessary this JIRA's to fix (nor is it introduced in this patch), but the manual URL concatenation strikes me as a bit ugly. Most web frameworks have utilities to add GET parameters and such and to build URLs for you. Not sure if one is handy in our environment, but usually more readable than long string concatenations.

          Tests

          There aren't any Understandable in a preliminary patch.

          Show
          Philip Zeyliger added a comment - Hi Owen, When does the delegation token that the namenode provides (redirectToRandomDataNode()) expire? A lot of websites pass security tokens via cookies, because GET parameters tend to get written down in referrer fields and such. So there's the potential that someone will get their hands on your token. Am I right that the token lets anyone read any data as if they were you? I'd be more comfortable if it were cookie based (though that implies that your datanodes and your namenode are in the same domain, which might not be workable), though I do see how GET is simpler. The web security part of me is also worried that this is liable to CSRF ( http://en.wikipedia.org/wiki/Cross-site_request_forgery ) attacks. The key there, I think, is to make sure that when the namenode is issuing tokens, it's absolutely confident that it's issuing them to someone who is asking for them. URLEncoder.encode("/", "UTF-8") Might be worthwhile to make this a constant. It's unlikely to change URL Creation This isn't necessary this JIRA's to fix (nor is it introduced in this patch), but the manual URL concatenation strikes me as a bit ugly. Most web frameworks have utilities to add GET parameters and such and to build URLs for you. Not sure if one is handy in our environment, but usually more readable than long string concatenations. Tests There aren't any Understandable in a preliminary patch.
          Hide
          Owen O'Malley added a comment -

          This patch is preliminary, but it covers the basics. It looks for a user parameter in the namenode and gets a delegation token for the redirect. All of the datanode operations use the delegation token to create the dfs client and access the filesystem.

          I also removed the static Configuration out of the JspHelper and set the configuration in an attribute in the datanode. (It was already being done in the name node.)

          Show
          Owen O'Malley added a comment - This patch is preliminary, but it covers the basics. It looks for a user parameter in the namenode and gets a delegation token for the redirect. All of the datanode operations use the delegation token to create the dfs client and access the filesystem. I also removed the static Configuration out of the JspHelper and set the configuration in an attribute in the datanode. (It was already being done in the name node.)

            People

            • Assignee:
              Owen O'Malley
              Reporter:
              Owen O'Malley
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development