Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3251

Network ACLs can prevent some clients to talk to MR ApplicationMaster

    Details

    • Type: Task
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.1
    • Component/s: mrv2
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      In 0.20.xxx, the JobClient while polling goes to JT to get the job status. With YARN, AM can be launched on any port and the client will have to have ACL open to that port to talk to AM and get the job status. When the client is within the same grid network access to AM is not a problem. But some applications may have one installation per set of clusters and may launch jobs even across such sets (on job trackers in another set of clusters). For that to work only the JT port needs to be open currently. In case of YARN, all ports will have to be opened up for things to work. That would be a security no-no.

      There are two possible solutions:
      1) Make the job client only talk to RM (as an option) to get the job status.
      2) Limit the range of ports AM can listen on.

      Option 2) may not be favorable as there is no direct OS API to find a free port.

        Attachments

        1. MAPREDUCE-3251-branch_0_23.patch
          13 kB
          Anupam Seth
        2. MAPREDUCE-3251-branch_0_23.patch
          14 kB
          Anupam Seth
        3. MAPREDUCE-3251-branch_0_23.patch
          15 kB
          Anupam Seth
        4. MAPREDUCE-3251-branch_0_23.patch
          16 kB
          Mahadev konar
        5. MAPREDUCE-3251-branch_0_23_incremental_fix.patch
          1 kB
          Anupam Seth
        6. MAPREDUCE-3251-branch_0_23_incremental_fix_2.patch
          11 kB
          Anupam Seth
        7. MAPREDUCE-3251-20120110.txt
          14 kB
          Vinod Kumar Vavilapalli
        8. MAPREDUCE-3251_branch-0_23_preliminary.txt
          4 kB
          Anupam Seth

          Activity

            People

            • Assignee:
              anupamseth Anupam Seth
              Reporter:
              anupamseth Anupam Seth
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: