Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6889

Add Job#close API to shutdown MR client services.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-beta1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      ATS1.5 uses FileSystemTimelineWriter which creates FS object on every writer initialization. If writer is not closed, then there is possibility of OOM see YARN-5438 fixes closing FS object.

      TimelineClient is used by YarnClient. So all the user who uses YarnClient with ATS1.5 need to stop service properly. Otherwise there is big chance of FS object leak.

      Of course MR uses YARN client submit job. If MR do not stop YarnClient then there is FS object leak.

      JobClient provides a API to stop all these service using JobClient#close. But many MR clients uses Job object to submit a job. But do not stop started services by default.

      So, provide a API Job#close to shutdown MR client services. This API can be utilized by caller explicitly to stop service which avoids FS leak.

      1. MAPREDUCE-6889.0001.patch
        1 kB
        Rohith Sharma K S
      2. MAPREDUCE-6889.0002.patch
        2 kB
        Rohith Sharma K S

        Activity

        Hide
        rohithsharma Rohith Sharma K S added a comment -

        Updated the patch with exposing public API.

        Show
        rohithsharma Rohith Sharma K S added a comment - Updated the patch with exposing public API.
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        Pseudo code to use this API would be

        // Create a new Job
        Job job = Job.getInstance();
        job.setJarByClass(MyJob.class);
        
        // Specify various job-specific parameters     
        job.setJobName("myjob");
        
        job.setInputPath(new Path("in"));
        job.setOutputPath(new Path("out"));
        
        job.setMapperClass(MyJob.MyMapper.class);
        job.setReducerClass(MyJob.MyReducer.class);
        
        // Submit the job, then poll for progress until the job is complete
        job.waitForCompletion(true);
        
        // Close the Job
        job.close();
        
        Show
        rohithsharma Rohith Sharma K S added a comment - Pseudo code to use this API would be // Create a new Job Job job = Job.getInstance(); job.setJarByClass(MyJob.class); // Specify various job-specific parameters job.setJobName( "myjob" ); job.setInputPath( new Path( "in" )); job.setOutputPath( new Path( "out" )); job.setMapperClass(MyJob.MyMapper.class); job.setReducerClass(MyJob.MyReducer.class); // Submit the job, then poll for progress until the job is complete job.waitForCompletion( true ); // Close the Job job.close();
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 36s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 13m 2s trunk passed
        +1 compile 0m 23s trunk passed
        +1 checkstyle 0m 17s trunk passed
        +1 mvnsite 0m 25s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 0m 47s trunk passed
        +1 javadoc 0m 19s trunk passed
        +1 mvninstall 0m 23s the patch passed
        +1 compile 0m 19s the patch passed
        +1 javac 0m 19s the patch passed
        +1 checkstyle 0m 13s the patch passed
        +1 mvnsite 0m 22s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 0m 50s the patch passed
        -1 javadoc 0m 15s hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core generated 1 new + 2496 unchanged - 0 fixed = 2497 total (was 2496)
        +1 unit 2m 33s hadoop-mapreduce-client-core in the patch passed.
        +1 asflicense 0m 16s The patch does not generate ASF License warnings.
        21m 58s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12868019/MAPREDUCE-6889.0001.patch
        JIRA Issue MAPREDUCE-6889
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 93052bf8ddff 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 6600abb
        Default Java 1.8.0_121
        findbugs v3.1.0-RC1
        javadoc https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6979/artifact/patchprocess/diff-javadoc-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
        Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6979/testReport/
        modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6979/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 36s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 13m 2s trunk passed +1 compile 0m 23s trunk passed +1 checkstyle 0m 17s trunk passed +1 mvnsite 0m 25s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 47s trunk passed +1 javadoc 0m 19s trunk passed +1 mvninstall 0m 23s the patch passed +1 compile 0m 19s the patch passed +1 javac 0m 19s the patch passed +1 checkstyle 0m 13s the patch passed +1 mvnsite 0m 22s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 50s the patch passed -1 javadoc 0m 15s hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core generated 1 new + 2496 unchanged - 0 fixed = 2497 total (was 2496) +1 unit 2m 33s hadoop-mapreduce-client-core in the patch passed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 21m 58s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12868019/MAPREDUCE-6889.0001.patch JIRA Issue MAPREDUCE-6889 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 93052bf8ddff 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6600abb Default Java 1.8.0_121 findbugs v3.1.0-RC1 javadoc https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6979/artifact/patchprocess/diff-javadoc-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6979/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6979/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        cc Jason Lowe would you please share your opinion this JIRA patch?

        Show
        rohithsharma Rohith Sharma K S added a comment - cc Jason Lowe would you please share your opinion this JIRA patch?
        Hide
        jlowe Jason Lowe added a comment -

        Seems like a reasonable approach. It would be good to have a unit test that verifies closing the job closes the cluster (or at least the client protocol provider). Also should cluster be set to null after we call close on it?

        Also this change alone isn't sufficient to fix the problem described in the JIRA summary. Callers would have to be updated to invoke the new close method at the appropriate time, so the headline is a bit misleading if we marked this as fixed.

        Show
        jlowe Jason Lowe added a comment - Seems like a reasonable approach. It would be good to have a unit test that verifies closing the job closes the cluster (or at least the client protocol provider). Also should cluster be set to null after we call close on it? Also this change alone isn't sufficient to fix the problem described in the JIRA summary. Callers would have to be updated to invoke the new close method at the appropriate time, so the headline is a bit misleading if we marked this as fixed.
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        It would be good to have a unit test that verifies closing the job closes the cluster (or at least the client protocol provider). Also should cluster be set to null after we call close on it?

        make sense, I will add a test case.

        Callers would have to be updated to invoke the new close method at the appropriate time, so the headline is a bit misleading if we marked this as fixed.

        thats true. Caller must close the Job object to avoid leak. I will also update JIRA summary.

        Show
        rohithsharma Rohith Sharma K S added a comment - It would be good to have a unit test that verifies closing the job closes the cluster (or at least the client protocol provider). Also should cluster be set to null after we call close on it? make sense, I will add a test case. Callers would have to be updated to invoke the new close method at the appropriate time, so the headline is a bit misleading if we marked this as fixed. thats true. Caller must close the Job object to avoid leak. I will also update JIRA summary.
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        Just for curiosity, many folks including me get general doubt that, why FileSystemTimelineWriter did not implement FileSystem#get() that uses cache rather than FileSystem#newInstance() which creates new object?
        One reason I would think Is for making independent TimelineClient so that closing FS object in JVM should not impact TimelineClient. Is my thought right?

        Show
        rohithsharma Rohith Sharma K S added a comment - Just for curiosity, many folks including me get general doubt that, why FileSystemTimelineWriter did not implement FileSystem#get() that uses cache rather than FileSystem#newInstance() which creates new object? One reason I would think Is for making independent TimelineClient so that closing FS object in JVM should not impact TimelineClient. Is my thought right?
        Hide
        jlowe Jason Lowe added a comment -

        I don't know the full history behind the newInstance call. Some software archaeology shows the newInstance change was added during the review rework process for YARN-4696, but even before that the code was explicitly setting the conf key to disable caching for the filesystem. The disabling of the filesystem cache was done partway through the review on YARN-4234 but I could not find a discussion that explained why that addition to the patch was necessary. Pinging Xuan Gong and Li Lu to see if they know the logic behind avoiding the fs cache in FileSystemTimelineWriter.

        Show
        jlowe Jason Lowe added a comment - I don't know the full history behind the newInstance call. Some software archaeology shows the newInstance change was added during the review rework process for YARN-4696 , but even before that the code was explicitly setting the conf key to disable caching for the filesystem. The disabling of the filesystem cache was done partway through the review on YARN-4234 but I could not find a discussion that explained why that addition to the patch was necessary. Pinging Xuan Gong and Li Lu to see if they know the logic behind avoiding the fs cache in FileSystemTimelineWriter.
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        Updated patch fixing review comments.

        Show
        rohithsharma Rohith Sharma K S added a comment - Updated patch fixing review comments.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 20s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 9s Maven dependency ordering for branch
        +1 mvninstall 14m 17s trunk passed
        +1 compile 2m 7s trunk passed
        +1 checkstyle 0m 33s trunk passed
        +1 mvnsite 1m 4s trunk passed
        +1 mvneclipse 0m 37s trunk passed
        +1 findbugs 1m 27s trunk passed
        +1 javadoc 0m 33s trunk passed
        0 mvndep 0m 7s Maven dependency ordering for patch
        +1 mvninstall 0m 57s the patch passed
        +1 compile 2m 0s the patch passed
        +1 javac 2m 0s the patch passed
        +1 checkstyle 0m 28s the patch passed
        +1 mvnsite 1m 1s the patch passed
        +1 mvneclipse 0m 33s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 46s the patch passed
        +1 javadoc 0m 33s the patch passed
        +1 unit 2m 53s hadoop-mapreduce-client-core in the patch passed.
        -1 unit 114m 49s hadoop-mapreduce-client-jobclient in the patch failed.
        +1 asflicense 0m 28s The patch does not generate ASF License warnings.
        147m 38s



        Reason Tests
        Timed out junit tests org.apache.hadoop.mapreduce.lib.jobcontrol.TestMapReduceJobControl



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12868968/MAPREDUCE-6889.0002.patch
        JIRA Issue MAPREDUCE-6889
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 5641169a87da 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 009b9f3
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6982/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
        unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6982/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
        Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6982/testReport/
        modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6982/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 9s Maven dependency ordering for branch +1 mvninstall 14m 17s trunk passed +1 compile 2m 7s trunk passed +1 checkstyle 0m 33s trunk passed +1 mvnsite 1m 4s trunk passed +1 mvneclipse 0m 37s trunk passed +1 findbugs 1m 27s trunk passed +1 javadoc 0m 33s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 0m 57s the patch passed +1 compile 2m 0s the patch passed +1 javac 2m 0s the patch passed +1 checkstyle 0m 28s the patch passed +1 mvnsite 1m 1s the patch passed +1 mvneclipse 0m 33s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 46s the patch passed +1 javadoc 0m 33s the patch passed +1 unit 2m 53s hadoop-mapreduce-client-core in the patch passed. -1 unit 114m 49s hadoop-mapreduce-client-jobclient in the patch failed. +1 asflicense 0m 28s The patch does not generate ASF License warnings. 147m 38s Reason Tests Timed out junit tests org.apache.hadoop.mapreduce.lib.jobcontrol.TestMapReduceJobControl Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12868968/MAPREDUCE-6889.0002.patch JIRA Issue MAPREDUCE-6889 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 5641169a87da 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 009b9f3 Default Java 1.8.0_131 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6982/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6982/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6982/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6982/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        sunilg Sunil G added a comment -

        +1 for the patch. Thanks for correct the jira title as well.
        I could commit tomorrow if there are no objections.

        Show
        sunilg Sunil G added a comment - +1 for the patch. Thanks for correct the jira title as well. I could commit tomorrow if there are no objections.
        Hide
        sunilg Sunil G added a comment -

        Committing shortly.

        Show
        sunilg Sunil G added a comment - Committing shortly.
        Hide
        sunilg Sunil G added a comment -

        Thanks Rohith Sharma K S. Committed to trunk/branch-2. and thanks Jason Lowe for the reviews.

        Show
        sunilg Sunil G added a comment - Thanks Rohith Sharma K S . Committed to trunk/branch-2. and thanks Jason Lowe for the reviews.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12018 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12018/)
        MAPREDUCE-6889. Add Job#close API to shutdown MR client services. (sunilg: rev fb3b5d33ffb29ee8e1ffbd2eee7a603a5777ebaf)

        • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
        • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMapperReducerCleanup.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12018 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12018/ ) MAPREDUCE-6889 . Add Job#close API to shutdown MR client services. (sunilg: rev fb3b5d33ffb29ee8e1ffbd2eee7a603a5777ebaf) (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMapperReducerCleanup.java

          People

          • Assignee:
            rohithsharma Rohith Sharma K S
            Reporter:
            rohithsharma Rohith Sharma K S
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development