Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6889

Add Job#close API to shutdown MR client services.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-beta1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      ATS1.5 uses FileSystemTimelineWriter which creates FS object on every writer initialization. If writer is not closed, then there is possibility of OOM see YARN-5438 fixes closing FS object.

      TimelineClient is used by YarnClient. So all the user who uses YarnClient with ATS1.5 need to stop service properly. Otherwise there is big chance of FS object leak.

      Of course MR uses YARN client submit job. If MR do not stop YarnClient then there is FS object leak.

      JobClient provides a API to stop all these service using JobClient#close. But many MR clients uses Job object to submit a job. But do not stop started services by default.

      So, provide a API Job#close to shutdown MR client services. This API can be utilized by caller explicitly to stop service which avoids FS leak.

        Attachments

        1. MAPREDUCE-6889.0001.patch
          1 kB
          Rohith Sharma K S
        2. MAPREDUCE-6889.0002.patch
          2 kB
          Rohith Sharma K S

          Activity

            People

            • Assignee:
              rohithsharma Rohith Sharma K S
              Reporter:
              rohithsharma Rohith Sharma K S
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: