Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6607

Enable regex pattern matching when mapreduce.task.files.preserve.filepattern is set

    Details

    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Before this fix, the files in .staging directory are always preserved when mapreduce.task.files.preserve.filepattern is set. After this fix, the files in .staging directory are preserved if the name of the directory matches the regex pattern specified by mapreduce.task.files.preserve.filepattern.
      Show
      Before this fix, the files in .staging directory are always preserved when mapreduce.task.files.preserve.filepattern is set. After this fix, the files in .staging directory are preserved if the name of the directory matches the regex pattern specified by mapreduce.task.files.preserve.filepattern.

      Description

      if either of the following configs are set, then .staging dir is not cleaned up:

      • mapreduce.task.files.preserve.failedtask
      • mapreduce.task.files.preserve.filepattern

      The former was supposed to keep only .staging of failed tasks and the latter was supposed to be used only if that task name matches against the specified regular expression.

        protected boolean keepJobFiles(JobConf conf) {
          return (conf.getKeepTaskFilesPattern() != null || conf
              .getKeepFailedTaskFiles());
        }
      
        public void cleanupStagingDir() throws IOException {
          /* make sure we clean the staging files */
          String jobTempDir = null;
          FileSystem fs = getFileSystem(getConfig());
          try {
            if (!keepJobFiles(new JobConf(getConfig()))) {
              jobTempDir = getConfig().get(MRJobConfig.MAPREDUCE_JOB_DIR);
              if (jobTempDir == null) {
                LOG.warn("Job Staging directory is null");
                return;
              }
              Path jobTempDirPath = new Path(jobTempDir);
              LOG.info("Deleting staging directory " + FileSystem.getDefaultUri(getConfig()) +
                  " " + jobTempDir);
              fs.delete(jobTempDirPath, true);
            }
          } catch(IOException io) {
            LOG.error("Failed to cleanup staging dir " + jobTempDir, io);
          }
        }
      

        Attachments

        1. MAPREDUCE-6607.01.patch
          7 kB
          Kai Sasaki
        2. MAPREDUCE-6607.02.patch
          7 kB
          Kai Sasaki
        3. MAPREDUCE-6607.03.patch
          8 kB
          Kai Sasaki
        4. MAPREDUCE-6607.04.patch
          8 kB
          Kai Sasaki
        5. MAPREDUCE-6607.05.patch
          10 kB
          Kai Sasaki
        6. MAPREDUCE-6607.06.patch
          10 kB
          Kai Sasaki
        7. MAPREDUCE-6607-branch-2.01.patch
          10 kB
          Kai Sasaki
        8. MAPREDUCE-6607-branch-2.02.patch
          10 kB
          Kai Sasaki

          Activity

            People

            • Assignee:
              lewuathe Kai Sasaki
              Reporter:
              maysamyabandeh Maysam Yabandeh
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: