Hadoop Common
  1. Hadoop Common
  2. HADOOP-7661

FileSystem.getCanonicalServiceName throws NPE for any file system uri that doesn't have an authority.

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.20.205.0
    • Fix Version/s: 0.20.205.0
    • Component/s: None
    • Labels:
      None

      Description

      FileSystem.getCanonicalServiceName throws NPE for any file system uri that doesn't have an authority.

      ....
      java.lang.NullPointerException
      at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:138)
      at org.apache.hadoop.security.SecurityUtil.buildDTServiceName(SecurityUtil.java:261)
      at org.apache.hadoop.fs.FileSystem.getCanonicalServiceName(FileSystem.java:174)
      ....

      1. HADOOP-7661.20s.3.patch
        4 kB
        Jitendra Nath Pandey
      2. HADOOP-7661.20s.2.patch
        3 kB
        Jitendra Nath Pandey
      3. HADOOP-7661.20s.1.patch
        3 kB
        Jitendra Nath Pandey

        Activity

        Hide
        Mit Desai added a comment -

        My bad. I did not see any commit message above or any comment saying someone will check in the changes.
        I see that the patch is committed to 0.20.205. Thanks for pointing this Ravi Prakash.

        Show
        Mit Desai added a comment - My bad. I did not see any commit message above or any comment saying someone will check in the changes. I see that the patch is committed to 0.20.205. Thanks for pointing this Ravi Prakash .
        Ravi Prakash made changes -
        Fix Version/s 0.20.205.0 [ 12316390 ]
        Hide
        Ravi Prakash added a comment -

        Mit! You shouldn't remove the Fix Version. This patch HAS gone into 0.20.205

        Show
        Ravi Prakash added a comment - Mit! You shouldn't remove the Fix Version. This patch HAS gone into 0.20.205
        Mit Desai made changes -
        Fix Version/s 0.20.205.0 [ 12316390 ]
        Mit Desai made changes -
        Target Version/s 0.20.205.0, 0.23.0 [ 12316390, 12315569 ] 0.20.205.0, 3.0.0 [ 12316390, 12320357 ]
        Hide
        Mit Desai added a comment -

        I did not hear anything since last couple of days. 0.23 is going into maintenance mode. Re-targeting this to 3.0.0. Also removing the fix version. It should be modified once the patch is committed.

        Show
        Mit Desai added a comment - I did not hear anything since last couple of days. 0.23 is going into maintenance mode. Re-targeting this to 3.0.0. Also removing the fix version. It should be modified once the patch is committed.
        Hide
        Mit Desai added a comment -

        This Jira hasn't been updated since a long time. Are we still experiencing this NPE?

        Show
        Mit Desai added a comment - This Jira hasn't been updated since a long time. Are we still experiencing this NPE?
        Matt Foley made changes -
        Target Version/s 0.20.205.0, 0.23.0 [ 12316390, 12315569 ]
        Hide
        Matt Foley added a comment -

        Jitendra, as part of this bug please commit the unit test for NPE on local file system, to trunk, so we'll know if it ever recurs there. Thanks.

        Show
        Matt Foley added a comment - Jitendra, as part of this bug please commit the unit test for NPE on local file system, to trunk, so we'll know if it ever recurs there. Thanks.
        Hide
        John George added a comment -

        Why not? From a cursory examination of the code, it looks like trunk should have the exact same issue.

        MAPREDUCE-2780 changed the behavior of buildDTServiceName() which started throwing an NPE for filesystems without valid authority. Since MAPREDUCE-2780 is not in trunk yet, the behavior is not there yet. The fix was to let FileSystem without a valid URI override the getCannonocalServiceName() eg: HADOOP-7602

        Show
        John George added a comment - Why not? From a cursory examination of the code, it looks like trunk should have the exact same issue. MAPREDUCE-2780 changed the behavior of buildDTServiceName() which started throwing an NPE for filesystems without valid authority. Since MAPREDUCE-2780 is not in trunk yet, the behavior is not there yet. The fix was to let FileSystem without a valid URI override the getCannonocalServiceName() eg: HADOOP-7602
        Hide
        Aaron T. Myers added a comment -

        Trunk doesn't have this NPE problem.

        Why not? From a cursory examination of the code, it looks like trunk should have the exact same issue.

        Show
        Aaron T. Myers added a comment - Trunk doesn't have this NPE problem. Why not? From a cursory examination of the code, it looks like trunk should have the exact same issue.
        Jitendra Nath Pandey made changes -
        Attachment HADOOP-7661.20s.3.patch [ 12495426 ]
        Hide
        Jitendra Nath Pandey added a comment -

        The patch includes a test for local file system just to make sure NPE is not being seen.

        Show
        Jitendra Nath Pandey added a comment - The patch includes a test for local file system just to make sure NPE is not being seen.
        Hide
        Jitendra Nath Pandey added a comment -

        It seems to me that this issue (and also HADOOP-7602) got introduced in MAPREDUCE-2780. I would prefer that the trunk patch for that jira, doesn't introduce this issue or the issue in HADOOP-7602. We should port the tests though.

        Show
        Jitendra Nath Pandey added a comment - It seems to me that this issue (and also HADOOP-7602 ) got introduced in MAPREDUCE-2780 . I would prefer that the trunk patch for that jira, doesn't introduce this issue or the issue in HADOOP-7602 . We should port the tests though.
        Hide
        John George added a comment -

        Jitendra, if HADOOP-7602 goes to trunk, would it be good to put this change in as well for consistency?

        Show
        John George added a comment - Jitendra, if HADOOP-7602 goes to trunk, would it be good to put this change in as well for consistency?
        Hide
        Suresh Srinivas added a comment -

        +1 for the patch. If there are no unit tests for this functionality, we should add one.

        Can you please create a trunk jira to annotate this method as private and stable. Stable since changing this would be backward incompatible.

        Show
        Suresh Srinivas added a comment - +1 for the patch. If there are no unit tests for this functionality, we should add one. Can you please create a trunk jira to annotate this method as private and stable. Stable since changing this would be backward incompatible.
        Jitendra Nath Pandey made changes -
        Attachment HADOOP-7661.20s.2.patch [ 12495324 ]
        Hide
        Jitendra Nath Pandey added a comment -

        Javadoc updated addressing Suresh's comment.

        Show
        Jitendra Nath Pandey added a comment - Javadoc updated addressing Suresh's comment.
        Hide
        Jitendra Nath Pandey added a comment -

        Trunk doesn't have this NPE problem. But, I agree with Suresh to annotate this method as private (to hadoop) in trunk.

        Show
        Jitendra Nath Pandey added a comment - Trunk doesn't have this NPE problem. But, I agree with Suresh to annotate this method as private (to hadoop) in trunk.
        Hide
        Aaron T. Myers added a comment -

        Does this change not also need to be made on trunk?

        Show
        Aaron T. Myers added a comment - Does this change not also need to be made on trunk?
        Hide
        Suresh Srinivas added a comment -

        Can you please add javadoc to the FileSystem method indicating the distinction between local file systems with no authority and remote file systems with authority and the expected behavior. We should also consider annotating this method as private in trunk.

        Show
        Suresh Srinivas added a comment - Can you please add javadoc to the FileSystem method indicating the distinction between local file systems with no authority and remote file systems with authority and the expected behavior. We should also consider annotating this method as private in trunk.
        Jitendra Nath Pandey made changes -
        Field Original Value New Value
        Attachment HADOOP-7661.20s.1.patch [ 12495287 ]
        Hide
        Jitendra Nath Pandey added a comment -

        This patch makes following change:
        The default behavior for getCanonicalServiceName is to just return the uri as the string, because calling SecurityUtil.buildDTServiceName doesn't make sense for many file systems. DFS and Hftp override getCanonicalServiceName to call buildDTServiceName

        Show
        Jitendra Nath Pandey added a comment - This patch makes following change: The default behavior for getCanonicalServiceName is to just return the uri as the string, because calling SecurityUtil.buildDTServiceName doesn't make sense for many file systems. DFS and Hftp override getCanonicalServiceName to call buildDTServiceName
        Jitendra Nath Pandey created issue -

          People

          • Assignee:
            Jitendra Nath Pandey
            Reporter:
            Jitendra Nath Pandey
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:

              Development