Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-7933

Viewfs changes for MAPREDUCE-3529

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 0.23.0
    • 0.23.1
    • viewfs
    • None
    • Reviewed

    Description

      ViewFs.getDelegationTokens returns a list of tokens for the associated namenodes. Credentials serializes these tokens using the service name for the actual namenodes. Effectively, tokens are not cached for viewfs (some more details in MR 3529). Affects any job which uses the TokenCache in tasks along with viewfs (some Pig jobs).

      Talk to Jitendra about this, some options
      1. Change Credentials.getAllTokens to return the key, instead of just a token list (associate the viewfs canonical name with a token in credentials)
      2. Have viewfs issue a fake token.
      Both of these would allow for a single viewfs configuration only.
      3. An additional API in FileSystem - something like getDelegationTokens(String renewer, Credentials credentials) - which would check the credentials object before making token requests to the actual namenode.
      4. An additional API in FileSystem - getCanonicalServiceNames - similar to getDelegationTokens, which would return service names for the actual namenodes. TokenCache/Credentials can work using this list.
      5. have getDelegationTokens check the current UGI - and fetch tokens only if they don't exist.

      Have a quick patch for 3, along with associated MR changes.

      Attachments

        1. HDFS2665_v1.txt
          14 kB
          Siddharth Seth
        2. HDFS2665_v1.txt
          14 kB
          Siddharth Seth
        3. HADOOP-7933.txt
          10 kB
          Suresh Srinivas
        4. HADOOP7933_v1.txt
          11 kB
          Siddharth Seth
        5. HADOOP7933_v2.txt
          11 kB
          Siddharth Seth

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            sseth Siddharth Seth
            sseth Siddharth Seth
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Issue deployment