Hadoop Common
  1. Hadoop Common
  2. HADOOP-7967

Need generalized multi-token filesystem support

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.23.1, 0.24.0
    • Fix Version/s: 0.23.3, 2.0.2-alpha
    • Component/s: fs, security
    • Labels:
      None

      Description

      Multi-token filesystem support and its interactions with the MR TokenCache is problematic. The TokenCache tries to assume it has the knowledge to know if the tokens for a filesystem are available, which it can't possibly know for multi-token filesystems. Filtered filesystems are also problematic, such as har on viewfs. When mergeFs is implemented, it too will become a problem with the current implementation. Currently FileSystem will leak tokens even when some tokens are already present.

      The decision for token acquisition, and which tokens, should be pushed all the way down into the FileSystem level. The TokenCache should be ignorant and simply request tokens from each FileSystem.

      1. HADOOP-7967.newapi.5.patch
        88 kB
        Daryn Sharp
      2. HADOOP-7967.newapi.4.patch
        77 kB
        Daryn Sharp
      3. hadoop7967-deltas.patch
        9 kB
        Sanjay Radia
      4. HADOOP-7967.newapi.3.patch
        88 kB
        Daryn Sharp
      5. hadoop7967-javadoc.patch
        3 kB
        Sanjay Radia
      6. HADOOP-7967.newapi.2.patch
        52 kB
        Daryn Sharp
      7. HADOOP-7967.newapi.patch
        44 kB
        Daryn Sharp
      8. HADOOP-7967-compat.patch
        19 kB
        Daryn Sharp
      9. HADOOP-7967-4.patch
        12 kB
        Daryn Sharp
      10. HADOOP-7967-3.patch
        12 kB
        Daryn Sharp
      11. HADOOP-7967-2.patch
        6 kB
        Daryn Sharp
      12. HADOOP-7967.patch
        11 kB
        Daryn Sharp

        Issue Links

          Activity

          Daryn Sharp created issue -
          Daryn Sharp made changes -
          Field Original Value New Value
          Attachment HADOOP-7967.patch [ 12510062 ]
          Daryn Sharp made changes -
          Link This issue is related to MAPREDUCE-3639 [ MAPREDUCE-3639 ]
          Daryn Sharp made changes -
          Link This issue is related to HDFS-2710 [ HDFS-2710 ]
          Daryn Sharp made changes -
          Link This issue is related to HADOOP-7933 [ HADOOP-7933 ]
          Daryn Sharp made changes -
          Assignee Daryn Sharp [ daryn ]
          Daryn Sharp made changes -
          Link This issue relates to HDFS-2854 [ HDFS-2854 ]
          Daryn Sharp made changes -
          Attachment HADOOP-7967-2.patch [ 12513522 ]
          Daryn Sharp made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Target Version/s 0.23.1, 0.24.0 [ 12318884, 12317652 ] 0.24.0, 0.23.1 [ 12317652, 12318884 ]
          Daryn Sharp made changes -
          Link This issue blocks MAPREDUCE-3825 [ MAPREDUCE-3825 ]
          Daryn Sharp made changes -
          Attachment HADOOP-7967-3.patch [ 12513632 ]
          Daryn Sharp made changes -
          Attachment HADOOP-7967-4.patch [ 12513643 ]
          Daryn Sharp made changes -
          Link This issue is duplicated by HDFS-2854 [ HDFS-2854 ]
          Daryn Sharp made changes -
          Link This issue relates to HADOOP-8048 [ HADOOP-8048 ]
          Arun C Murthy made changes -
          Target Version/s 0.23.1, 0.24.0 [ 12318884, 12317652 ] 0.23.2 [ 12319855 ]
          Daryn Sharp made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Robert Joseph Evans made changes -
          Target Version/s 0.23.2 [ 12319855 ] 0.23.3, 2.0.0, 3.0.0 [ 12320059, 12320352, 12320357 ]
          Daryn Sharp made changes -
          Attachment HADOOP-7967-compat.patch [ 12531026 ]
          Daryn Sharp made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Target Version/s 0.23.3, 2.0.0-alpha, 3.0.0 [ 12320059, 12320352, 12320357 ] 2.0.0-alpha, 0.23.3, 3.0.0 [ 12320352, 12320059, 12320357 ]
          Daryn Sharp made changes -
          Attachment HADOOP-7967-compat.patch [ 12531026 ]
          Daryn Sharp made changes -
          Attachment HADOOP-7967-compat.patch [ 12531113 ]
          Daryn Sharp made changes -
          Attachment HADOOP-7967.newapi.patch [ 12536854 ]
          Daryn Sharp made changes -
          Attachment HADOOP-7967.newapi.2.patch [ 12537603 ]
          Thomas Graves made changes -
          Target Version/s 0.23.3, 2.0.0-alpha, 3.0.0 [ 12320059, 12320352, 12320357 ] 2.0.0-alpha, 0.23.3, 3.0.0 [ 12320352, 12320059, 12320357 ]
          Priority Major [ 3 ] Critical [ 2 ]
          Sanjay Radia made changes -
          Attachment hadoop7967-javadoc.patch [ 12538428 ]
          Sanjay Radia made changes -
          Attachment hadoop7967-javadoc.patch [ 12538430 ]
          Sanjay Radia made changes -
          Attachment hadoop7967-javadoc.patch [ 12538428 ]
          Daryn Sharp made changes -
          Attachment HADOOP-7967.newapi.3.patch [ 12539093 ]
          Sanjay Radia made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Target Version/s 0.23.3, 2.0.0-alpha, 3.0.0 [ 12320059, 12320352, 12320357 ] 2.0.0-alpha, 0.23.3, 3.0.0 [ 12320352, 12320059, 12320357 ]
          Sanjay Radia made changes -
          Attachment hadoop7967-deltas.patch [ 12540120 ]
          Daryn Sharp made changes -
          Attachment HADOOP-7967.newapi.4.patch [ 12541059 ]
          Daryn Sharp made changes -
          Link This issue relates to HADOOP-8701 [ HADOOP-8701 ]
          Daryn Sharp made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Target Version/s 0.23.3, 2.0.0-alpha, 3.0.0 [ 12320059, 12320352, 12320357 ] 2.0.0-alpha, 0.23.3, 3.0.0 [ 12320352, 12320059, 12320357 ]
          Daryn Sharp made changes -
          Attachment HADOOP-7967.newapi.5.patch [ 12541137 ]
          Daryn Sharp made changes -
          Fix Version/s 0.23.3 [ 12320059 ]
          Fix Version/s 2.1.0-alpha [ 12321441 ]
          Fix Version/s 3.0.0 [ 12320357 ]
          Fix Version/s 2.2.0-alpha [ 12322473 ]
          Target Version/s 0.23.3, 2.0.0-alpha, 3.0.0 [ 12320059, 12320352, 12320357 ] 2.0.0-alpha, 0.23.3, 3.0.0 [ 12320352, 12320059, 12320357 ]
          Daryn Sharp made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Target Version/s 0.23.3, 2.0.0-alpha, 3.0.0 [ 12320059, 12320352, 12320357 ] 2.0.0-alpha, 0.23.3, 3.0.0 [ 12320352, 12320059, 12320357 ]
          Resolution Fixed [ 1 ]
          Arun C Murthy made changes -
          Fix Version/s 3.0.0 [ 12320357 ]
          Fix Version/s 2.1.0-alpha [ 12321441 ]
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Daryn Sharp
              Reporter:
              Daryn Sharp
            • Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development