Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 2.0.0-alpha
    • Component/s: ha, hdfs-client
    • Labels:
      None

      Description

      fs.getCanonicalService() must be equal to fs.getDelegationToken(renewer).getService(). When HA is enabled, the DFS token's service is a logical uri, but dfs.getCanonicalService() is only returning the hostname of the logical uri.

        Issue Links

          Activity

          Daryn Sharp created issue -
          Hide
          Daryn Sharp added a comment -

          getCanonicalServiceName returns HAUtil.buildTokenServiceForLogicalUri if HA is enabled.

          Show
          Daryn Sharp added a comment - getCanonicalServiceName returns HAUtil.buildTokenServiceForLogicalUri if HA is enabled.
          Daryn Sharp made changes -
          Field Original Value New Value
          Attachment HDFS-3255.patch [ 12522335 ]
          Daryn Sharp made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Todd Lipcon added a comment -

          hey Daryn. Fix looks good. Is there a manual test that be run with MR, for example, to verify this as well? i.e how did you discover this issue?

          Show
          Todd Lipcon added a comment - hey Daryn. Fix looks good. Is there a manual test that be run with MR, for example, to verify this as well? i.e how did you discover this issue?
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12522335/HDFS-3255.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2253//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2253//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522335/HDFS-3255.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2253//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2253//console This message is automatically generated.
          Hide
          Daryn Sharp added a comment -

          I found the problem while reviewing the code for host-based tokens and then confirmed it was a problem via the unit tests. Proper handling of the service is very important. The service is used to determine whether a token is already acquired.

          I can see if I can use Jitendra's feature to enable security for a unit test if you'd like.

          Show
          Daryn Sharp added a comment - I found the problem while reviewing the code for host-based tokens and then confirmed it was a problem via the unit tests. Proper handling of the service is very important. The service is used to determine whether a token is already acquired. I can see if I can use Jitendra's feature to enable security for a unit test if you'd like.
          Hide
          Todd Lipcon added a comment -

          Hm, I'm just surprised, because it seemed to work when we tested running MR against an HA cluster. But maybe there's some bug that happens when it comes time to renew the token, or something?

          I can see if I can use Jitendra's feature to enable security for a unit test if you'd like.

          When I tried to use that feature, I couldn't get it to work. Maybe you'll have better luck?

          If you can describe a manual test scenario that seems good enough for me.

          Show
          Todd Lipcon added a comment - Hm, I'm just surprised, because it seemed to work when we tested running MR against an HA cluster. But maybe there's some bug that happens when it comes time to renew the token, or something? I can see if I can use Jitendra's feature to enable security for a unit test if you'd like. When I tried to use that feature, I couldn't get it to work. Maybe you'll have better luck? If you can describe a manual test scenario that seems good enough for me.
          Hide
          Daryn Sharp added a comment -

          I should clarify: It "works" but multiple tokens are fetched for the HA filesystem. The reason is the TokenCache builds a map of tokens keyed on their service. The value from fs.getCanonicalServiceName() is used to check the map for a pre-existing token. Since the canonical service doesn't match the value of token.getService(), it repeatedly fetches a token for a HA filesystem.

          Show
          Daryn Sharp added a comment - I should clarify: It "works" but multiple tokens are fetched for the HA filesystem. The reason is the TokenCache builds a map of tokens keyed on their service. The value from fs.getCanonicalServiceName() is used to check the map for a pre-existing token. Since the canonical service doesn't match the value of token.getService() , it repeatedly fetches a token for a HA filesystem.
          Hide
          Daryn Sharp added a comment -

          A manual test would be to run a job with multiple input paths. DFSClient logs an info message when a token is fetched, so log should show only one token retrieved.

          Show
          Daryn Sharp added a comment - A manual test would be to run a job with multiple input paths. DFSClient logs an info message when a token is fetched, so log should show only one token retrieved.
          Hide
          Todd Lipcon added a comment -

          k, sounds good. +1

          Show
          Todd Lipcon added a comment - k, sounds good. +1
          Hide
          Todd Lipcon added a comment -

          Committed to branch-2 and trunk. Thanks, Daryn!

          Show
          Todd Lipcon added a comment - Committed to branch-2 and trunk. Thanks, Daryn!
          Todd Lipcon made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Fix Version/s 2.0.0 [ 12320353 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2137 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2137/)
          HDFS-3255. HA DFS returns wrong token service. Contributed by Daryn Sharp. (Revision 1325414)

          Result = SUCCESS
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1325414
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDelegationTokensWithHA.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2137 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2137/ ) HDFS-3255 . HA DFS returns wrong token service. Contributed by Daryn Sharp. (Revision 1325414) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1325414 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDelegationTokensWithHA.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2063 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2063/)
          HDFS-3255. HA DFS returns wrong token service. Contributed by Daryn Sharp. (Revision 1325414)

          Result = SUCCESS
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1325414
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDelegationTokensWithHA.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2063 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2063/ ) HDFS-3255 . HA DFS returns wrong token service. Contributed by Daryn Sharp. (Revision 1325414) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1325414 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDelegationTokensWithHA.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2077 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2077/)
          HDFS-3255. HA DFS returns wrong token service. Contributed by Daryn Sharp. (Revision 1325414)

          Result = SUCCESS
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1325414
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDelegationTokensWithHA.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2077 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2077/ ) HDFS-3255 . HA DFS returns wrong token service. Contributed by Daryn Sharp. (Revision 1325414) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1325414 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDelegationTokensWithHA.java
          Daryn Sharp made changes -
          Link This issue relates to HDFS-3268 [ HDFS-3268 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1013 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1013/)
          HDFS-3255. HA DFS returns wrong token service. Contributed by Daryn Sharp. (Revision 1325414)

          Result = FAILURE
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1325414
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDelegationTokensWithHA.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1013 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1013/ ) HDFS-3255 . HA DFS returns wrong token service. Contributed by Daryn Sharp. (Revision 1325414) Result = FAILURE todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1325414 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDelegationTokensWithHA.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1048 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1048/)
          HDFS-3255. HA DFS returns wrong token service. Contributed by Daryn Sharp. (Revision 1325414)

          Result = SUCCESS
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1325414
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDelegationTokensWithHA.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1048 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1048/ ) HDFS-3255 . HA DFS returns wrong token service. Contributed by Daryn Sharp. (Revision 1325414) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1325414 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDelegationTokensWithHA.java

            People

            • Assignee:
              Daryn Sharp
              Reporter:
              Daryn Sharp
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development