Hadoop Common
  1. Hadoop Common
  2. HADOOP-6579

A utility for reading and writing tokens into a URL safe string.

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: security
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      We need to include HDFS delegation tokens in the URLs while browsing the file system. Therefore, we need a url-safe way to encode and decode them.

      1. c-6579.patch
        9 kB
        Owen O'Malley
      2. c-6579.patch
        9 kB
        Owen O'Malley
      3. c-6579.patch
        8 kB
        Owen O'Malley
      4. c-6579.patch
        5 kB
        Owen O'Malley
      5. c-6579.patch
        3 kB
        Owen O'Malley
      6. c-6579-hdfs.patch
        2 kB
        Owen O'Malley
      7. c-6579-mr.patch
        1 kB
        Owen O'Malley

        Issue Links

          Activity

          Hide
          Owen O'Malley added a comment -

          This patch adds encode/decode methods into WritableUtils and ups the version of commons-code from 1.3 to 1.4. The patches for hdfs and mr are trivial.

          Show
          Owen O'Malley added a comment - This patch adds encode/decode methods into WritableUtils and ups the version of commons-code from 1.3 to 1.4. The patches for hdfs and mr are trivial.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12436467/c-6579.patch
          against trunk revision 912207.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/22/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/22/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/22/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/22/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12436467/c-6579.patch against trunk revision 912207. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/22/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/22/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/22/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/22/console This message is automatically generated.
          Hide
          Owen O'Malley added a comment -

          Included a test case.

          Show
          Owen O'Malley added a comment - Included a test case.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12436787/c-6579.patch
          against trunk revision 915168.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/374/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/374/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/374/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/374/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12436787/c-6579.patch against trunk revision 915168. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/374/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/374/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/374/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/374/console This message is automatically generated.
          Hide
          Philip Zeyliger added a comment -

          Hi Owen,

          Trying to keep up with some of the security jiras. You're producing a lot of code, thereby making it tricky

          I think, in general, it's not that useful to write arbitrary writables to base64-encoded strings. Most browsers limit how long URL strings can be, so you've got to be pretty careful about what you're up to. Would you consider instead making this more specific, by moving this code into Token.getAsUrlSafeString() and (static) Token.fromUrlSafeString()? Or, equivalently, leave the code here, but in redirectToRandomDataNode() (patch in HDFS-991), use a method on the Token instead of WritableUtils. (This has the additional property that one could serialize tokens however; they just have to have a URL-safe string serialization.)

          Looked at the code and tests. Those look clear and good.

          – Philip

          Show
          Philip Zeyliger added a comment - Hi Owen, Trying to keep up with some of the security jiras. You're producing a lot of code, thereby making it tricky I think, in general, it's not that useful to write arbitrary writables to base64-encoded strings. Most browsers limit how long URL strings can be, so you've got to be pretty careful about what you're up to. Would you consider instead making this more specific, by moving this code into Token.getAsUrlSafeString() and (static) Token.fromUrlSafeString()? Or, equivalently, leave the code here, but in redirectToRandomDataNode() (patch in HDFS-991 ), use a method on the Token instead of WritableUtils. (This has the additional property that one could serialize tokens however; they just have to have a URL-safe string serialization.) Looked at the code and tests. Those look clear and good. – Philip
          Hide
          Owen O'Malley added a comment -

          I think it is more useful than that, but I've made the change to more the methods into Token. I also added toString, equals, hashCode, and new constructor to Token so that I could test the new code.

          Show
          Owen O'Malley added a comment - I think it is more useful than that, but I've made the change to more the methods into Token. I also added toString, equals, hashCode, and new constructor to Token so that I could test the new code.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12436908/c-6579.patch
          against trunk revision 915168.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/379/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/379/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/379/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/379/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12436908/c-6579.patch against trunk revision 915168. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/379/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/379/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/379/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/379/console This message is automatically generated.
          Hide
          Devaraj Das added a comment -

          +1 (except that the test failure needs to be checked)

          Show
          Devaraj Das added a comment - +1 (except that the test failure needs to be checked)
          Hide
          Owen O'Malley added a comment -

          This patch fixes one of the test cases to add the extra parameters to the Base64 constructor. It must have changed semantics between 1.3 and 1.4 of the library.

          Show
          Owen O'Malley added a comment - This patch fixes one of the test cases to add the extra parameters to the Base64 constructor. It must have changed semantics between 1.3 and 1.4 of the library.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12436955/c-6579.patch
          against trunk revision 915168.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The patch appears to cause tar ant target to fail.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/381/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/381/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/381/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/381/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12436955/c-6579.patch against trunk revision 915168. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The patch appears to cause tar ant target to fail. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/381/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/381/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/381/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/381/console This message is automatically generated.
          Hide
          Owen O'Malley added a comment -

          I missed one file in the previous patch.

          Show
          Owen O'Malley added a comment - I missed one file in the previous patch.
          Hide
          Devaraj Das added a comment -

          +1 (subject to hudson)

          Show
          Devaraj Das added a comment - +1 (subject to hudson)
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12436959/c-6579.patch
          against trunk revision 915168.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/382/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/382/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/382/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/382/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12436959/c-6579.patch against trunk revision 915168. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/382/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/382/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/382/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/382/console This message is automatically generated.
          Hide
          Owen O'Malley added a comment -

          I just committed this.

          Show
          Owen O'Malley added a comment - I just committed this.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #203 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/203/)
          . Upgrade the commons-codec library to 1.4. (omalley)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #203 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/203/ ) . Upgrade the commons-codec library to 1.4. (omalley)
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk #261 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/261/)
          . Provide a mechanism for encoding/decoding Tokens from
          a url-safe string and change the commons-code library to 1.4. (omalley)

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk #261 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/261/ ) . Provide a mechanism for encoding/decoding Tokens from a url-safe string and change the commons-code library to 1.4. (omalley)
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #252 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/252/)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #252 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/252/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #193 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/193/)
          . Provide a mechanism for encoding/decoding Tokens from
          a url-safe string and change the commons-code library to 1.4. (omalley)

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #193 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/193/ ) . Provide a mechanism for encoding/decoding Tokens from a url-safe string and change the commons-code library to 1.4. (omalley)
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #146 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/146/)

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #146 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/146/ )
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #302 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/302/)

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #302 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/302/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #275 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/275/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #275 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/275/ )

            People

            • Assignee:
              Owen O'Malley
              Reporter:
              Owen O'Malley
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development