[MAPREDUCE-6621] Memory Leak in JobClient#submitJobInternal()

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.8.0, 2.7.3, 2.6.4, 3.0.0-alpha1
• Target Version/s: 2.6.4
• Hadoop Flags: Reviewed
• Assignee: Xuan Gong
• Reporter: Xuan Gong
• Created: 29/Jan/16 01:47
• Updated: 06/Jan/17 01:47
• Resolved: 02/Feb/16 19:34

Description

In JobClient:

public RunningJob submitJobInternal(final JobConf conf)
      throws FileNotFoundException, IOException {
    try {
      conf.setBooleanIfUnset("mapred.mapper.new-api", false);
      conf.setBooleanIfUnset("mapred.reducer.new-api", false);
      Job job = clientUgi.doAs(new PrivilegedExceptionAction<Job> () {
        @Override
        public Job run() throws IOException, ClassNotFoundException, 
          InterruptedException {
          Job job = Job.getInstance(conf);
          job.submit();
          return job;
        }
      });

      // update our Cluster instance with the one created by Job for submission
      // (we can't pass our Cluster instance to Job, since Job wraps the config
      // instance, and the two configs would then diverge)
      cluster = job.getCluster();

      return new NetworkedJob(job);
    } catch (InterruptedException ie) {
      throw new IOException("interrupted", ie);
    }
  }

We will replace the cluster object with the cluster object from Job, but the previous old cluster object would never be closed.

Issue Links

• is related to: YARN-5309 Fix SSLFactory truststore reloader thread leak in TimelineClientImpl
• is related to: MAPREDUCE-6618 YarnClientProtocolProvider leaking the YarnClient thread.