Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-7967

Need generalized multi-token filesystem support

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 0.23.1, 0.24.0
    • Fix Version/s: 0.23.3, 2.0.2-alpha
    • Component/s: fs, security
    • Labels:
      None

      Description

      Multi-token filesystem support and its interactions with the MR TokenCache is problematic. The TokenCache tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently FileSystem will leak tokens even when some tokens are already present.

      The decision for token acquisition, and which tokens, should be pushed all the way down into the FileSystem level. The TokenCache should be ignorant and simply request tokens from each FileSystem.

        Attachments

        1. HADOOP-7967.newapi.5.patch
          88 kB
          Daryn Sharp
        2. HADOOP-7967.newapi.4.patch
          77 kB
          Daryn Sharp
        3. hadoop7967-deltas.patch
          9 kB
          Sanjay Radia
        4. HADOOP-7967.newapi.3.patch
          88 kB
          Daryn Sharp
        5. hadoop7967-javadoc.patch
          3 kB
          Sanjay Radia
        6. HADOOP-7967.newapi.2.patch
          52 kB
          Daryn Sharp
        7. HADOOP-7967.newapi.patch
          44 kB
          Daryn Sharp
        8. HADOOP-7967-compat.patch
          19 kB
          Daryn Sharp
        9. HADOOP-7967-4.patch
          12 kB
          Daryn Sharp
        10. HADOOP-7967-3.patch
          12 kB
          Daryn Sharp
        11. HADOOP-7967-2.patch
          6 kB
          Daryn Sharp
        12. HADOOP-7967.patch
          11 kB
          Daryn Sharp

          Issue Links

            Activity

              People

              • Assignee:
                daryn Daryn Sharp
                Reporter:
                daryn Daryn Sharp
              • Votes:
                0 Vote for this issue
                Watchers:
                12 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: