Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6621

Memory Leak in JobClient#submitJobInternal()

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 2.7.3, 2.6.4, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      In JobClient:

      public RunningJob submitJobInternal(final JobConf conf)
            throws FileNotFoundException, IOException {
          try {
            conf.setBooleanIfUnset("mapred.mapper.new-api", false);
            conf.setBooleanIfUnset("mapred.reducer.new-api", false);
            Job job = clientUgi.doAs(new PrivilegedExceptionAction<Job> () {
              @Override
              public Job run() throws IOException, ClassNotFoundException, 
                InterruptedException {
                Job job = Job.getInstance(conf);
                job.submit();
                return job;
              }
            });
      
            // update our Cluster instance with the one created by Job for submission
            // (we can't pass our Cluster instance to Job, since Job wraps the config
            // instance, and the two configs would then diverge)
            cluster = job.getCluster();
      
            return new NetworkedJob(job);
          } catch (InterruptedException ie) {
            throw new IOException("interrupted", ie);
          }
        }
      

      We will replace the cluster object with the cluster object from Job, but the previous old cluster object would never be closed.

        Attachments

        1. MAPREDUCE-6621.1.patch
          2 kB
          Xuan Gong
        2. MAPREDUCE-6621.2.patch
          1 kB
          Xuan Gong

          Issue Links

            Activity

              People

              • Assignee:
                xgong Xuan Gong
                Reporter:
                xgong Xuan Gong
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: