Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.24.0, 0.23.1
    • Fix Version/s: 0.24.0, 0.23.1
    • Component/s: webhdfs
    • Labels:
      None

      Description

      Need to port 205 tokens into these filesystems. Will mainly involve ensuring code duplicated from hftp is updated accordingly.

      1. HDFS-2785.txt
        2 kB
        Robert Joseph Evans

        Activity

        Eli Collins made changes -
        Component/s webhdfs [ 12319200 ]
        Component/s name-node [ 12312926 ]
        Component/s security [ 12313400 ]
        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #979 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/979/)
        HDFS-2785. Update webhdfs and httpfs for host-based token support. Contributed by Robert Joseph Evans.

        jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240460
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #979 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/979/ ) HDFS-2785 . Update webhdfs and httpfs for host-based token support. Contributed by Robert Joseph Evans. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240460 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Build #181 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/181/)
        Merged r1240460 from trunk for HDFS-2785.

        jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240461
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #181 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/181/ ) Merged r1240460 from trunk for HDFS-2785 . jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240461 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #159 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/159/)
        Merged r1240460 from trunk for HDFS-2785.

        jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240461
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #159 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/159/ ) Merged r1240460 from trunk for HDFS-2785 . jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240461 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #946 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/946/)
        HDFS-2785. Update webhdfs and httpfs for host-based token support. Contributed by Robert Joseph Evans.

        jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240460
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #946 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/946/ ) HDFS-2785 . Update webhdfs and httpfs for host-based token support. Contributed by Robert Joseph Evans. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240460 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Commit #499 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/499/)
        Merged r1240460 from trunk for HDFS-2785.

        jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240461
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #499 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/499/ ) Merged r1240460 from trunk for HDFS-2785 . jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240461 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #1674 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1674/)
        HDFS-2785. Update webhdfs and httpfs for host-based token support. Contributed by Robert Joseph Evans.

        jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240460
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1674 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1674/ ) HDFS-2785 . Update webhdfs and httpfs for host-based token support. Contributed by Robert Joseph Evans. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240460 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-0.23-Commit #486 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/486/)
        Merged r1240460 from trunk for HDFS-2785.

        jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240461
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #486 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/486/ ) Merged r1240460 from trunk for HDFS-2785 . jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240461 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Commit #476 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/476/)
        Merged r1240460 from trunk for HDFS-2785.

        jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240461
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #476 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/476/ ) Merged r1240460 from trunk for HDFS-2785 . jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240461 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #1730 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1730/)
        HDFS-2785. Update webhdfs and httpfs for host-based token support. Contributed by Robert Joseph Evans.

        jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240460
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1730 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1730/ ) HDFS-2785 . Update webhdfs and httpfs for host-based token support. Contributed by Robert Joseph Evans. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240460 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #1659 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1659/)
        HDFS-2785. Update webhdfs and httpfs for host-based token support. Contributed by Robert Joseph Evans.

        jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240460
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1659 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1659/ ) HDFS-2785 . Update webhdfs and httpfs for host-based token support. Contributed by Robert Joseph Evans. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240460 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
        Jitendra Nath Pandey made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Target Version/s 0.23.1, 0.24.0 [ 12318885, 12317653 ] 0.24.0, 0.23.1 [ 12317653, 12318885 ]
        Fix Version/s 0.24.0 [ 12317653 ]
        Fix Version/s 0.23.1 [ 12318885 ]
        Resolution Fixed [ 1 ]
        Hide
        Jitendra Nath Pandey added a comment -

        I have committed this. Thanks to Robert.

        Show
        Jitendra Nath Pandey added a comment - I have committed this. Thanks to Robert.
        Hide
        Robert Joseph Evans added a comment -

        Jitendra, could you please check this in, I am not an HDFS committer, I am only a committer from common and mapred. I am filing the other JIRA now.

        Show
        Robert Joseph Evans added a comment - Jitendra, could you please check this in, I am not an HDFS committer, I am only a committer from common and mapred. I am filing the other JIRA now.
        Hide
        Jitendra Nath Pandey added a comment -

        I see, this means we should remove the method checkTGTAndReloginFromKeytab and only use reloginFromKeytab because it is redundant and confusing.

        No need to change the patch.
        Could you please file a jira to get rid of checkTGTAndReloginFromKeytab?

        +1 for the patch.

        Show
        Jitendra Nath Pandey added a comment - I see, this means we should remove the method checkTGTAndReloginFromKeytab and only use reloginFromKeytab because it is redundant and confusing. No need to change the patch. Could you please file a jira to get rid of checkTGTAndReloginFromKeytab? +1 for the patch.
        Hide
        Robert Joseph Evans added a comment -

        I agree that checkTGTAndReloginFromKeytab is not really necessary, and I am fine with switching it. However, If you look at UserGroupInformation.java on trunk checkTGTAndReloginFromKeytab is the following

        public synchronized void checkTGTAndReloginFromKeytab() throws IOException {
          if (!isSecurityEnabled()
              || user.getAuthenticationMethod() != AuthenticationMethod.KERBEROS
              || !isKeytab)
            return;
          KerberosTicket tgt = getTGT();
          if (tgt != null && System.currentTimeMillis() < getRefreshTime(tgt)) {
            return;
          }
          reloginFromKeytab();
        }
        

        But a few lines down on line 697 inside reloginFromKeytab() we have the exact same check.

        KerberosTicket tgt = getTGT();
        //Return if TGT is valid and is not going to expire soon.
        if (tgt != null && now < getRefreshTime(tgt)) {
          return;
        }
        

        and sorry if my explanation was confusing before about being consistent with HFTP. I made the change to be consistent with the version of HFTP from HDFS-2784, which was just merged to trunk. In that case getDelegationToken() and renew() call reloginFromKeytab(), and cancel() calls checkTGTAndReloginFromKeytab(). Which is what the attached patch causes WebHDFS to do as well.

        If you still want me to change it then please let me know so I can update the patch and rerun my tests.

        Show
        Robert Joseph Evans added a comment - I agree that checkTGTAndReloginFromKeytab is not really necessary, and I am fine with switching it. However, If you look at UserGroupInformation.java on trunk checkTGTAndReloginFromKeytab is the following public synchronized void checkTGTAndReloginFromKeytab() throws IOException { if (!isSecurityEnabled() || user.getAuthenticationMethod() != AuthenticationMethod.KERBEROS || !isKeytab) return ; KerberosTicket tgt = getTGT(); if (tgt != null && System .currentTimeMillis() < getRefreshTime(tgt)) { return ; } reloginFromKeytab(); } But a few lines down on line 697 inside reloginFromKeytab() we have the exact same check. KerberosTicket tgt = getTGT(); //Return if TGT is valid and is not going to expire soon. if (tgt != null && now < getRefreshTime(tgt)) { return ; } and sorry if my explanation was confusing before about being consistent with HFTP. I made the change to be consistent with the version of HFTP from HDFS-2784 , which was just merged to trunk. In that case getDelegationToken() and renew() call reloginFromKeytab(), and cancel() calls checkTGTAndReloginFromKeytab(). Which is what the attached patch causes WebHDFS to do as well. If you still want me to change it then please let me know so I can update the patch and rerun my tests.
        Hide
        Jitendra Nath Pandey added a comment -

        The last change in the patch i.e. changing from checkTGTAndReloginFromKeytab to reloginFromKeytab, doesn't seem necessary. The branch-1 code has checkTGTAndReloginFromKeytab and so does HftpFileSystem. I suppose WebHdfsFileSystem and HftpFileSystem should behave in the same way.

        I am sorry I should have raised this earlier and should have noticed the test results. Please ignore my previous comment.

        Show
        Jitendra Nath Pandey added a comment - The last change in the patch i.e. changing from checkTGTAndReloginFromKeytab to reloginFromKeytab, doesn't seem necessary. The branch-1 code has checkTGTAndReloginFromKeytab and so does HftpFileSystem. I suppose WebHdfsFileSystem and HftpFileSystem should behave in the same way. I am sorry I should have raised this earlier and should have noticed the test results. Please ignore my previous comment.
        Hide
        Robert Joseph Evans added a comment -

        I am a bit confused about updating the test results. The unit tests all passed with no changes to them at all. The test is very small and intended to be 100% backwards compatible. It made a few things consistent between WebHDFS and HFTP, and it change a small amount of URI processing changes where we now get data directly from the URI instead of getting it from INetSocetAddress.

        Show
        Robert Joseph Evans added a comment - I am a bit confused about updating the test results. The unit tests all passed with no changes to them at all. The test is very small and intended to be 100% backwards compatible. It made a few things consistent between WebHDFS and HFTP, and it change a small amount of URI processing changes where we now get data directly from the URI instead of getting it from INetSocetAddress.
        Hide
        Jitendra Nath Pandey added a comment -

        Please update with test results so that we can commit.

        Show
        Jitendra Nath Pandey added a comment - Please update with test results so that we can commit.
        Hide
        Jitendra Nath Pandey added a comment -

        +1

        Show
        Jitendra Nath Pandey added a comment - +1
        Hide
        Robert Joseph Evans added a comment -

        I filed HDFS-2836 for the javadoc warnings

        Show
        Robert Joseph Evans added a comment - I filed HDFS-2836 for the javadoc warnings
        Hide
        Robert Joseph Evans added a comment -

        The following are the warning that javadocs is exposing.

        [WARNING] org/apache/hadoop/fs/FileSystem.class(org/apache/hadoop/fs:FileSystem.class): warning: Cannot find annotation method 'value()' in type 'org.apache.hadoop.classification.InterfaceAudience.LimitedPrivate': class file for org.apache.hadoop.classification.InterfaceAudience not found
        [WARNING] org/apache/hadoop/fs/FileSystem.class(org/apache/hadoop/fs:FileSystem.class): warning: Cannot find annotation method 'value()' in type 'org.apache.hadoop.classification.InterfaceAudience.LimitedPrivate'
        [WARNING] org/apache/hadoop/fs/FileSystem.class(org/apache/hadoop/fs:FileSystem.class): warning: Cannot find annotation method 'value()' in type 'org.apache.hadoop.classification.InterfaceAudience.LimitedPrivate'
        [WARNING] org/apache/hadoop/security/Groups.class(org/apache/hadoop/security:Groups.class): warning: Cannot find annotation method 'value()' in type 'org.apache.hadoop.classification.InterfaceAudience.LimitedPrivate'
        [WARNING] org/apache/hadoop/security/UserGroupInformation.class(org/apache/hadoop/security:UserGroupInformation.class): warning: Cannot find annotation method 'value()' in type 'org.apache.hadoop.classification.InterfaceAudience.LimitedPrivate'
        [WARNING] org/apache/hadoop/security/UserGroupInformation.class(org/apache/hadoop/security:UserGroupInformation.class): warning: Cannot find annotation method 'value()' in type 'org.apache.hadoop.classification.InterfaceAudience.LimitedPrivate'
        [WARNING] org/apache/hadoop/security/UserGroupInformation.class(org/apache/hadoop/security:UserGroupInformation.class): warning: Cannot find annotation method 'value()' in type 'org.apache.hadoop.classification.InterfaceAudience.LimitedPrivate'
        [WARNING] org/apache/hadoop/fs/FSDataInputStream.class(org/apache/hadoop/fs:FSDataInputStream.class): warning: Cannot find annotation method 'value()' in type 'org.apache.hadoop.classification.InterfaceAudience.LimitedPrivate'
        [WARNING] org/apache/hadoop/fs/FSDataOutputStream.class(org/apache/hadoop/fs:FSDataOutputStream.class): warning: Cannot find annotation method 'value()' in type 'org.apache.hadoop.classification.InterfaceAudience.LimitedPrivate'
        [WARNING] org/apache/hadoop/fs/Path.class(org/apache/hadoop/fs:Path.class): warning: Cannot find annotation method 'value()' in type 'org.apache.hadoop.classification.InterfaceAudience.LimitedPrivate'
        [WARNING] org/apache/hadoop/fs/UnresolvedLinkException.class(org/apache/hadoop/fs:UnresolvedLinkException.class): warning: Cannot find annotation method 'value()' in type 'org.apache.hadoop.classification.InterfaceAudience.LimitedPrivate'
        [WARNING] org/apache/hadoop/fs/MD5MD5CRC32FileChecksum.class(org/apache/hadoop/fs:MD5MD5CRC32FileChecksum.class): warning: Cannot find annotation method 'value()' in type 'org.apache.hadoop.classification.InterfaceAudience.LimitedPrivate'
        [WARNING] org/apache/hadoop/fs/LocalDirAllocator.class(org/apache/hadoop/fs:LocalDirAllocator.class): warning: Cannot find annotation method 'value()' in type 'org.apache.hadoop.classification.InterfaceAudience.LimitedPrivate'
        [WARNING] org/apache/hadoop/fs/FileContext.class(org/apache/hadoop/fs:FileContext.class): warning: Cannot find annotation method 'value()' in type 'org.apache.hadoop.classification.InterfaceAudience.LimitedPrivate'
        [WARNING] org/apache/hadoop/fs/FileContext.class(org/apache/hadoop/fs:FileContext.class): warning: Cannot find annotation method 'value()' in type 'org.apache.hadoop.classification.InterfaceAudience.LimitedPrivate'
        [WARNING] org/apache/hadoop/fs/FSOutputSummer.class(org/apache/hadoop/fs:FSOutputSummer.class): warning: Cannot find annotation method 'value()' in type 'org.apache.hadoop.classification.InterfaceAudience.LimitedPrivate'
        [WARNING] org/apache/hadoop/fs/FSInputStream.class(org/apache/hadoop/fs:FSInputStream.class): warning: Cannot find annotation method 'value()' in type 'org.apache.hadoop.classification.InterfaceAudience.LimitedPrivate'
        [WARNING] org/apache/hadoop/fs/FSInputChecker.class(org/apache/hadoop/fs:FSInputChecker.class): warning: Cannot find annotation method 'value()' in type 'org.apache.hadoop.classification.InterfaceAudience.LimitedPrivate'
        [WARNING] org/apache/hadoop/fs/DU.class(org/apache/hadoop/fs:DU.class): warning: Cannot find annotation method 'value()' in type 'org.apache.hadoop.classification.InterfaceAudience.LimitedPrivate'
        [WARNING] org/apache/hadoop/fs/DF.class(org/apache/hadoop/fs:DF.class): warning: Cannot find annotation method 'value()' in type 'org.apache.hadoop.classification.InterfaceAudience.LimitedPrivate'
        [WARNING] org/apache/hadoop/fs/AbstractFileSystem.class(org/apache/hadoop/fs:AbstractFileSystem.class): warning: Cannot find annotation method 'value()' in type 'org.apache.hadoop.classification.InterfaceAudience.LimitedPrivate'
        [WARNING] hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSServer.java:241: warning - @param argument "override," is not a parameter name.
        [WARNING] hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSServer.java:450: warning - @param argument "override," is not a parameter name.
        

        None of these are related to any files I have modified. I will look through them and file JIRAS for any ones that I cannot find a JIRA for.

        Show
        Robert Joseph Evans added a comment - The following are the warning that javadocs is exposing. [WARNING] org/apache/hadoop/fs/FileSystem.class(org/apache/hadoop/fs:FileSystem.class): warning: Cannot find annotation method 'value()' in type 'org.apache.hadoop.classification.InterfaceAudience.LimitedPrivate': class file for org.apache.hadoop.classification.InterfaceAudience not found [WARNING] org/apache/hadoop/fs/FileSystem.class(org/apache/hadoop/fs:FileSystem.class): warning: Cannot find annotation method 'value()' in type 'org.apache.hadoop.classification.InterfaceAudience.LimitedPrivate' [WARNING] org/apache/hadoop/fs/FileSystem.class(org/apache/hadoop/fs:FileSystem.class): warning: Cannot find annotation method 'value()' in type 'org.apache.hadoop.classification.InterfaceAudience.LimitedPrivate' [WARNING] org/apache/hadoop/security/Groups.class(org/apache/hadoop/security:Groups.class): warning: Cannot find annotation method 'value()' in type 'org.apache.hadoop.classification.InterfaceAudience.LimitedPrivate' [WARNING] org/apache/hadoop/security/UserGroupInformation.class(org/apache/hadoop/security:UserGroupInformation.class): warning: Cannot find annotation method 'value()' in type 'org.apache.hadoop.classification.InterfaceAudience.LimitedPrivate' [WARNING] org/apache/hadoop/security/UserGroupInformation.class(org/apache/hadoop/security:UserGroupInformation.class): warning: Cannot find annotation method 'value()' in type 'org.apache.hadoop.classification.InterfaceAudience.LimitedPrivate' [WARNING] org/apache/hadoop/security/UserGroupInformation.class(org/apache/hadoop/security:UserGroupInformation.class): warning: Cannot find annotation method 'value()' in type 'org.apache.hadoop.classification.InterfaceAudience.LimitedPrivate' [WARNING] org/apache/hadoop/fs/FSDataInputStream.class(org/apache/hadoop/fs:FSDataInputStream.class): warning: Cannot find annotation method 'value()' in type 'org.apache.hadoop.classification.InterfaceAudience.LimitedPrivate' [WARNING] org/apache/hadoop/fs/FSDataOutputStream.class(org/apache/hadoop/fs:FSDataOutputStream.class): warning: Cannot find annotation method 'value()' in type 'org.apache.hadoop.classification.InterfaceAudience.LimitedPrivate' [WARNING] org/apache/hadoop/fs/Path.class(org/apache/hadoop/fs:Path.class): warning: Cannot find annotation method 'value()' in type 'org.apache.hadoop.classification.InterfaceAudience.LimitedPrivate' [WARNING] org/apache/hadoop/fs/UnresolvedLinkException.class(org/apache/hadoop/fs:UnresolvedLinkException.class): warning: Cannot find annotation method 'value()' in type 'org.apache.hadoop.classification.InterfaceAudience.LimitedPrivate' [WARNING] org/apache/hadoop/fs/MD5MD5CRC32FileChecksum.class(org/apache/hadoop/fs:MD5MD5CRC32FileChecksum.class): warning: Cannot find annotation method 'value()' in type 'org.apache.hadoop.classification.InterfaceAudience.LimitedPrivate' [WARNING] org/apache/hadoop/fs/LocalDirAllocator.class(org/apache/hadoop/fs:LocalDirAllocator.class): warning: Cannot find annotation method 'value()' in type 'org.apache.hadoop.classification.InterfaceAudience.LimitedPrivate' [WARNING] org/apache/hadoop/fs/FileContext.class(org/apache/hadoop/fs:FileContext.class): warning: Cannot find annotation method 'value()' in type 'org.apache.hadoop.classification.InterfaceAudience.LimitedPrivate' [WARNING] org/apache/hadoop/fs/FileContext.class(org/apache/hadoop/fs:FileContext.class): warning: Cannot find annotation method 'value()' in type 'org.apache.hadoop.classification.InterfaceAudience.LimitedPrivate' [WARNING] org/apache/hadoop/fs/FSOutputSummer.class(org/apache/hadoop/fs:FSOutputSummer.class): warning: Cannot find annotation method 'value()' in type 'org.apache.hadoop.classification.InterfaceAudience.LimitedPrivate' [WARNING] org/apache/hadoop/fs/FSInputStream.class(org/apache/hadoop/fs:FSInputStream.class): warning: Cannot find annotation method 'value()' in type 'org.apache.hadoop.classification.InterfaceAudience.LimitedPrivate' [WARNING] org/apache/hadoop/fs/FSInputChecker.class(org/apache/hadoop/fs:FSInputChecker.class): warning: Cannot find annotation method 'value()' in type 'org.apache.hadoop.classification.InterfaceAudience.LimitedPrivate' [WARNING] org/apache/hadoop/fs/DU.class(org/apache/hadoop/fs:DU.class): warning: Cannot find annotation method 'value()' in type 'org.apache.hadoop.classification.InterfaceAudience.LimitedPrivate' [WARNING] org/apache/hadoop/fs/DF.class(org/apache/hadoop/fs:DF.class): warning: Cannot find annotation method 'value()' in type 'org.apache.hadoop.classification.InterfaceAudience.LimitedPrivate' [WARNING] org/apache/hadoop/fs/AbstractFileSystem.class(org/apache/hadoop/fs:AbstractFileSystem.class): warning: Cannot find annotation method 'value()' in type 'org.apache.hadoop.classification.InterfaceAudience.LimitedPrivate' [WARNING] hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSServer.java:241: warning - @param argument "override," is not a parameter name. [WARNING] hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSServer.java:450: warning - @param argument "override," is not a parameter name. None of these are related to any files I have modified. I will look through them and file JIRAS for any ones that I cannot find a JIRA for.
        Hide
        Robert Joseph Evans added a comment -

        The find bugs issue is unrelated, but I filed HDFS-2835 to fix it. I am looking into the javadoc warnings, but as I changed no javadocs and no APIs I am fairly sure they are unrelated. I already explained why there are no new tests added.

        Show
        Robert Joseph Evans added a comment - The find bugs issue is unrelated, but I filed HDFS-2835 to fix it. I am looking into the javadoc warnings, but as I changed no javadocs and no APIs I am fairly sure they are unrelated. I already explained why there are no new tests added.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12511699/HDFS-2785.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 javadoc. The javadoc tool appears to have generated 21 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1804//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1804//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1804//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12511699/HDFS-2785.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated 21 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1804//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1804//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1804//console This message is automatically generated.
        Robert Joseph Evans made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Target Version/s 0.23.1, 0.24.0 [ 12318885, 12317653 ] 0.24.0, 0.23.1 [ 12317653, 12318885 ]
        Robert Joseph Evans made changes -
        Field Original Value New Value
        Attachment HDFS-2785.txt [ 12511699 ]
        Hide
        Robert Joseph Evans added a comment -

        There are no tests included with this patch, because in reality none of the real functionality should have changed. There were just 3 things modified.

        1. The URI returned by getUri is no longer based off of the INetAddress for the NameNode. This is because the host is resolved in the INetAddress and not in the origninal URI.
        2. The code to get the INetAddress from a token goes directly to security Utils and not through NetUtils. This is to make the token more opaque as it was originally designed to be.
        3. The TGT renewal no longer checks for expirey of the TGT. This change is only to make it consistent with HFTP, and is not truly needed. It can be removed if reviewers want it to be removed.
        Show
        Robert Joseph Evans added a comment - There are no tests included with this patch, because in reality none of the real functionality should have changed. There were just 3 things modified. The URI returned by getUri is no longer based off of the INetAddress for the NameNode. This is because the host is resolved in the INetAddress and not in the origninal URI. The code to get the INetAddress from a token goes directly to security Utils and not through NetUtils. This is to make the token more opaque as it was originally designed to be. The TGT renewal no longer checks for expirey of the TGT. This change is only to make it consistent with HFTP, and is not truly needed. It can be removed if reviewers want it to be removed.
        Hide
        Robert Joseph Evans added a comment -

        I was wrong about that. Most of the code is fine, but there are a few small things that need to be changed. I am testing my patch now and will upload it shortly.

        Show
        Robert Joseph Evans added a comment - I was wrong about that. Most of the code is fine, but there are a few small things that need to be changed. I am testing my patch now and will upload it shortly.
        Hide
        Robert Joseph Evans added a comment -

        From my first pass through the code there appear to be no changes needed. I am going to do another more in-depth look to be sure. But this might just be closed as "Not A Problem"

        Show
        Robert Joseph Evans added a comment - From my first pass through the code there appear to be no changes needed. I am going to do another more in-depth look to be sure. But this might just be closed as "Not A Problem"
        Daryn Sharp created issue -

          People

          • Assignee:
            Robert Joseph Evans
            Reporter:
            Daryn Sharp
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development