Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3251

Network ACLs can prevent some clients to talk to MR ApplicationMaster

    Details

    • Type: Task Task
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.1
    • Component/s: mrv2
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      In 0.20.xxx, the JobClient while polling goes to JT to get the job status. With YARN, AM can be launched on any port and the client will have to have ACL open to that port to talk to AM and get the job status. When the client is within the same grid network access to AM is not a problem. But some applications may have one installation per set of clusters and may launch jobs even across such sets (on job trackers in another set of clusters). For that to work only the JT port needs to be open currently. In case of YARN, all ports will have to be opened up for things to work. That would be a security no-no.

      There are two possible solutions:
      1) Make the job client only talk to RM (as an option) to get the job status.
      2) Limit the range of ports AM can listen on.

      Option 2) may not be favorable as there is no direct OS API to find a free port.

      1. MAPREDUCE-3251-branch_0_23.patch
        13 kB
        Anupam Seth
      2. MAPREDUCE-3251-branch_0_23.patch
        14 kB
        Anupam Seth
      3. MAPREDUCE-3251-branch_0_23.patch
        15 kB
        Anupam Seth
      4. MAPREDUCE-3251-branch_0_23.patch
        16 kB
        Mahadev konar
      5. MAPREDUCE-3251-branch_0_23_incremental_fix.patch
        1 kB
        Anupam Seth
      6. MAPREDUCE-3251-branch_0_23_incremental_fix_2.patch
        11 kB
        Anupam Seth
      7. MAPREDUCE-3251-20120110.txt
        14 kB
        Vinod Kumar Vavilapalli
      8. MAPREDUCE-3251_branch-0_23_preliminary.txt
        4 kB
        Anupam Seth

        Activity

        Anupam Seth created issue -
        Vinod Kumar Vavilapalli made changes -
        Field Original Value New Value
        Summary JobClient should have an option to only to talk to RM to get job status JobClient should have an option to only to talk to RM+HistoryServer to get job status
        Fix Version/s 0.23.0 [ 12315570 ]
        Priority Major [ 3 ] Blocker [ 1 ]
        Anupam Seth made changes -
        Assignee Anupam Seth [ anupamseth ]
        Arun C Murthy made changes -
        Priority Blocker [ 1 ] Critical [ 2 ]
        Arun C Murthy made changes -
        Fix Version/s 0.23.1 [ 12318883 ]
        Fix Version/s 0.23.0 [ 12315570 ]
        Vinod Kumar Vavilapalli made changes -
        Summary JobClient should have an option to only to talk to RM+HistoryServer to get job status Network ACLs can prevent some clients to talk to MR ApplicationMaster
        Arun C Murthy made changes -
        Target Version/s 0.23.1 [ 12318883 ]
        Anupam Seth made changes -
        Anupam Seth made changes -
        Attachment MAPREDUCE-3251-branch_0_23.patch [ 12505786 ]
        Anupam Seth made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Mahadev konar made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Anupam Seth made changes -
        Attachment MAPREDUCE-3251-branch_0_23.patch [ 12506153 ]
        Anupam Seth made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Mahadev konar made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Anupam Seth made changes -
        Attachment MAPREDUCE-3251-branch_0_23.patch [ 12506759 ]
        Anupam Seth made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Mahadev konar made changes -
        Attachment MAPREDUCE-3251-branch_0_23.patch [ 12507490 ]
        Mahadev konar made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Resolution Fixed [ 1 ]
        Vinod Kumar Vavilapalli made changes -
        Resolution Fixed [ 1 ]
        Status Resolved [ 5 ] Reopened [ 4 ]
        Anupam Seth made changes -
        Anupam Seth made changes -
        Status Reopened [ 4 ] Patch Available [ 10002 ]
        Vinod Kumar Vavilapalli made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Anupam Seth made changes -
        Anupam Seth made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Vinod Kumar Vavilapalli made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Vinod Kumar Vavilapalli made changes -
        Attachment MAPREDUCE-3251-20120110.txt [ 12510112 ]
        Vinod Kumar Vavilapalli made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Vinod Kumar Vavilapalli made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Anupam Seth
            Reporter:
            Anupam Seth
          • Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development