Hadoop Common
  1. Hadoop Common
  2. HADOOP-7967

Need generalized multi-token filesystem support


    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.23.1, 0.24.0
    • Fix Version/s: 0.23.3, 2.0.2-alpha
    • Component/s: fs, security
    • Labels:


      Multi-token filesystem support and its interactions with the MR TokenCache is problematic. The TokenCache tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently FileSystem will leak tokens even when some tokens are already present.

      The decision for token acquisition, and which tokens, should be pushed all the way down into the FileSystem level. The TokenCache should be ignorant and simply request tokens from each FileSystem.

      1. HADOOP-7967.newapi.2.patch
        52 kB
        Daryn Sharp
      2. HADOOP-7967.newapi.3.patch
        88 kB
        Daryn Sharp
      3. HADOOP-7967.newapi.4.patch
        77 kB
        Daryn Sharp
      4. HADOOP-7967.newapi.5.patch
        88 kB
        Daryn Sharp
      5. HADOOP-7967.newapi.patch
        44 kB
        Daryn Sharp
      6. HADOOP-7967.patch
        11 kB
        Daryn Sharp
      7. HADOOP-7967-2.patch
        6 kB
        Daryn Sharp
      8. HADOOP-7967-3.patch
        12 kB
        Daryn Sharp
      9. HADOOP-7967-4.patch
        12 kB
        Daryn Sharp
      10. HADOOP-7967-compat.patch
        19 kB
        Daryn Sharp
      11. hadoop7967-deltas.patch
        9 kB
        Sanjay Radia
      12. hadoop7967-javadoc.patch
        3 kB
        Sanjay Radia

        Issue Links


          No work has yet been logged on this issue.


            • Assignee:
              Daryn Sharp
              Daryn Sharp
            • Votes:
              0 Vote for this issue
              13 Start watching this issue


              • Created: