Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-7967

Need generalized multi-token filesystem support

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 0.23.1, 0.24.0
    • 0.23.3, 2.0.2-alpha
    • fs, security
    • None

    Description

      Multi-token filesystem support and its interactions with the MR TokenCache is problematic. The TokenCache tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently FileSystem will leak tokens even when some tokens are already present.

      The decision for token acquisition, and which tokens, should be pushed all the way down into the FileSystem level. The TokenCache should be ignorant and simply request tokens from each FileSystem.

      Attachments

        1. hadoop7967-javadoc.patch
          3 kB
          Sanjay Radia
        2. hadoop7967-deltas.patch
          9 kB
          Sanjay Radia
        3. HADOOP-7967-compat.patch
          19 kB
          Daryn Sharp
        4. HADOOP-7967-4.patch
          12 kB
          Daryn Sharp
        5. HADOOP-7967-3.patch
          12 kB
          Daryn Sharp
        6. HADOOP-7967-2.patch
          6 kB
          Daryn Sharp
        7. HADOOP-7967.patch
          11 kB
          Daryn Sharp
        8. HADOOP-7967.newapi.patch
          44 kB
          Daryn Sharp
        9. HADOOP-7967.newapi.5.patch
          88 kB
          Daryn Sharp
        10. HADOOP-7967.newapi.4.patch
          77 kB
          Daryn Sharp
        11. HADOOP-7967.newapi.3.patch
          88 kB
          Daryn Sharp
        12. HADOOP-7967.newapi.2.patch
          52 kB
          Daryn Sharp

        Issue Links

        blocks

        Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-3825 MR should not be getting duplicate tokens for a MR Job.

        • Major - Major loss of function.
        • Resolved
        is duplicated by

        Bug - A problem which impairs or prevents the functions of the product. HDFS-2854 SecurityUtil.buildTokenService returns java.net.UnknownHostException when using paths like viewfs://default/some/path

        • Major - Major loss of function.
        • Resolved
        is related to

        Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-3639 TokenCache likely broken for FileSystems which don't issue delegation tokens

        • Blocker - Blocks development and/or testing work, production could not run
        • Closed

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-7933 Viewfs changes for MAPREDUCE-3529

        • Critical - Crashes, loss of data, severe memory leak.
        • Closed

        Bug - A problem which impairs or prevents the functions of the product. HDFS-2710 HDFS part of MAPREDUCE-3529, HADOOP-7933

        • Critical - Crashes, loss of data, severe memory leak.
        • Closed
        relates to

        Bug - A problem which impairs or prevents the functions of the product. HDFS-2854 SecurityUtil.buildTokenService returns java.net.UnknownHostException when using paths like viewfs://default/some/path

        • Major - Major loss of function.
        • Resolved

        Improvement - An improvement or enhancement to an existing feature or task. HADOOP-8701 Reduce visibility of getDelegationToken

        • Major - Major loss of function.
        • Open

        Improvement - An improvement or enhancement to an existing feature or task. HADOOP-8048 Allow merging of Credentials

        • Major - Major loss of function.
        • Resolved

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            daryn Daryn Sharp Assign to me
            daryn Daryn Sharp
            Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment