Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6621

Memory Leak in JobClient#submitJobInternal()

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 2.8.0, 2.7.3, 2.6.4, 3.0.0-alpha1
    • None
    • None
    • Reviewed

    Description

      In JobClient:

      public RunningJob submitJobInternal(final JobConf conf)
            throws FileNotFoundException, IOException {
          try {
            conf.setBooleanIfUnset("mapred.mapper.new-api", false);
            conf.setBooleanIfUnset("mapred.reducer.new-api", false);
            Job job = clientUgi.doAs(new PrivilegedExceptionAction<Job> () {
              @Override
              public Job run() throws IOException, ClassNotFoundException, 
                InterruptedException {
                Job job = Job.getInstance(conf);
                job.submit();
                return job;
              }
            });
      
            // update our Cluster instance with the one created by Job for submission
            // (we can't pass our Cluster instance to Job, since Job wraps the config
            // instance, and the two configs would then diverge)
            cluster = job.getCluster();
      
            return new NetworkedJob(job);
          } catch (InterruptedException ie) {
            throw new IOException("interrupted", ie);
          }
        }
      

      We will replace the cluster object with the cluster object from Job, but the previous old cluster object would never be closed.

      Attachments

        1. MAPREDUCE-6621.1.patch
          2 kB
          Xuan Gong
        2. MAPREDUCE-6621.2.patch
          1 kB
          Xuan Gong

        Issue Links

          Activity

            People

              xgong Xuan Gong
              xgong Xuan Gong
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: