Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5367

Local jobs all use same local working directory

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.2.0
    • Fix Version/s: 1.3.0, 2.1.1-beta
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      This means that local jobs, even in different JVMs, can't run concurrently because they might delete each other's files during work directory setup.

      1. MAPREDUCE-5367.patch
        4 kB
        Sandy Ryza
      2. MAPREDUCE-5367-b1.patch
        0.7 kB
        Sandy Ryza
      3. MAPREDUCE-5367-b1-1.patch
        0.8 kB
        Sandy Ryza

        Issue Links

          Activity

          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12590327/MAPREDUCE-5367.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3817//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3817//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12590327/MAPREDUCE-5367.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3817//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3817//console This message is automatically generated.
          Hide
          sandyr Sandy Ryza added a comment -

          Oops submitted patch to the wrong JIRA.

          Show
          sandyr Sandy Ryza added a comment - Oops submitted patch to the wrong JIRA.
          Hide
          bpodgursky Ben Podgursky added a comment -

          For what it's worth, our current hacky workaround to this bug is prefixing the working directory with a UUID in LocalJobRunner:

          String tmpDir = jobDir + "/" + id + "-"+ UUID.randomUUID();
          this.localJobDir = localFs.makeQualified(conf.getLocalPath(tmpDir));

          and deleting it on job cleanup:

          localFs.delete(localJobDir, true);

          But I'm sure there's a cleaner way to scope the paths.

          Show
          bpodgursky Ben Podgursky added a comment - For what it's worth, our current hacky workaround to this bug is prefixing the working directory with a UUID in LocalJobRunner: String tmpDir = jobDir + "/" + id + "-"+ UUID.randomUUID(); this.localJobDir = localFs.makeQualified(conf.getLocalPath(tmpDir)); and deleting it on job cleanup: localFs.delete(localJobDir, true); But I'm sure there's a cleaner way to scope the paths.
          Hide
          sandyr Sandy Ryza added a comment -

          My patch was going to just add the job ID in. Is there a reason that the random UUID is needed on top of that?

          Show
          sandyr Sandy Ryza added a comment - My patch was going to just add the job ID in. Is there a reason that the random UUID is needed on top of that?
          Hide
          bpodgursky Ben Podgursky added a comment -

          Yeah, the problem is if the jobs are running in different JVMs, the job numbering starts at over at 0001, so there are still conflicts (for example if our build server starts two tests concurrently.)

          Show
          bpodgursky Ben Podgursky added a comment - Yeah, the problem is if the jobs are running in different JVMs, the job numbering starts at over at 0001, so there are still conflicts (for example if our build server starts two tests concurrently.)
          Hide
          sandyr Sandy Ryza added a comment -

          This was fixed MAPREDUCE-4278, which adds a random number to the job ID.

          Show
          sandyr Sandy Ryza added a comment - This was fixed MAPREDUCE-4278 , which adds a random number to the job ID.
          Hide
          bpodgursky Ben Podgursky added a comment -

          Oh awesome, didn't know that was fixed. UUID is definitely unnecessary then.

          Show
          bpodgursky Ben Podgursky added a comment - Oh awesome, didn't know that was fixed. UUID is definitely unnecessary then.
          Hide
          sandyr Sandy Ryza added a comment -

          Uploading a patch that adds the job ID to the local job dir. As with MAPREDUCE-4278, it's difficult to write a test for this, because it requires two separate LocalJobRunner processes to reproduce the issue. I've run the existing LocalJobRunner tests as well as run concurrent jobs locally and verified that they use different directories.

          Show
          sandyr Sandy Ryza added a comment - Uploading a patch that adds the job ID to the local job dir. As with MAPREDUCE-4278 , it's difficult to write a test for this, because it requires two separate LocalJobRunner processes to reproduce the issue. I've run the existing LocalJobRunner tests as well as run concurrent jobs locally and verified that they use different directories.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12593094/MAPREDUCE-5367-b1.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3887//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12593094/MAPREDUCE-5367-b1.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3887//console This message is automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12593094/MAPREDUCE-5367-b1.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3890//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12593094/MAPREDUCE-5367-b1.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3890//console This message is automatically generated.
          Hide
          tomwhite Tom White added a comment -

          I noticed that jobDir is being assigned to LocalJobRunner.SUBDIR, i.e. "localRunner/" with no job ID, so getLocalTaskDir() will return a path that is not unique for the job. The fix here is to make getLocalTaskDir() non-static. If jobDir were renamed to JOB_BASE_DIR then this problem would be less likely to be re-introduced in the future.

          Also, jobDir ends in "/", so there is no need to add another path separator. It's probably better to use the new Path(parent, child) constructor anyway.

          Show
          tomwhite Tom White added a comment - I noticed that jobDir is being assigned to LocalJobRunner.SUBDIR , i.e. "localRunner/" with no job ID, so getLocalTaskDir() will return a path that is not unique for the job. The fix here is to make getLocalTaskDir() non-static. If jobDir were renamed to JOB_BASE_DIR then this problem would be less likely to be re-introduced in the future. Also, jobDir ends in "/", so there is no need to add another path separator. It's probably better to use the new Path(parent, child) constructor anyway.
          Hide
          sandyr Sandy Ryza added a comment -

          Are you looking at branch-1 or trunk? The patch is for branch-1, but only trunk's LocalJobRunner has SUBDIR and getLocalTaskDir.

          Will fix the double path separator.

          Show
          sandyr Sandy Ryza added a comment - Are you looking at branch-1 or trunk? The patch is for branch-1, but only trunk's LocalJobRunner has SUBDIR and getLocalTaskDir. Will fix the double path separator.
          Hide
          tomwhite Tom White added a comment -

          I was looking at trunk. Doesn't this need fixing for trunk too?

          Show
          tomwhite Tom White added a comment - I was looking at trunk. Doesn't this need fixing for trunk too?
          Hide
          sandyr Sandy Ryza added a comment -

          I don't think the problem exists in trunk. getLocalTaskDir includes the job ID in the path, so there shouldn't be collisions. The other place that localRunner/ is used is for writing the job conf, which includes the job ID in its name. So that also should not be a problem. Though thinking about it now, it might make sense to change it as well for consistency?

          Show
          sandyr Sandy Ryza added a comment - I don't think the problem exists in trunk. getLocalTaskDir includes the job ID in the path, so there shouldn't be collisions. The other place that localRunner/ is used is for writing the job conf, which includes the job ID in its name. So that also should not be a problem. Though thinking about it now, it might make sense to change it as well for consistency?
          Hide
          tomwhite Tom White added a comment -

          I think it would be worth changing for consistency.

          Show
          tomwhite Tom White added a comment - I think it would be worth changing for consistency.
          Hide
          sandyr Sandy Ryza added a comment -

          Uploading patches for both MR1 and MR2. For both, the job directory will be localRunner/user/job_id and the task directories will be localRunner/user/job_id/task_id.

          Show
          sandyr Sandy Ryza added a comment - Uploading patches for both MR1 and MR2. For both, the job directory will be localRunner/user/job_id and the task directories will be localRunner/user/job_id/task_id.
          Hide
          sandyr Sandy Ryza added a comment -

          Verified both with TestLocalRunner and local pi jobs.

          Show
          sandyr Sandy Ryza added a comment - Verified both with TestLocalRunner and local pi jobs.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12595060/MAPREDUCE-5367.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3917//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3917//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12595060/MAPREDUCE-5367.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3917//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3917//console This message is automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12595060/MAPREDUCE-5367.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3918//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3918//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12595060/MAPREDUCE-5367.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3918//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3918//console This message is automatically generated.
          Hide
          tomwhite Tom White added a comment -

          +1

          Show
          tomwhite Tom White added a comment - +1
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #4211 (See https://builds.apache.org/job/Hadoop-trunk-Commit/4211/)
          MAPREDUCE-5367. Local jobs all use same local working directory (Sandy Ryza) (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1510610)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalJobRunner.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #4211 (See https://builds.apache.org/job/Hadoop-trunk-Commit/4211/ ) MAPREDUCE-5367 . Local jobs all use same local working directory (Sandy Ryza) (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1510610 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalJobRunner.java
          Hide
          sandyr Sandy Ryza added a comment -

          Thanks for the review, Tom. Committed to branch-1, branch-2, branch-2.1-beta, and trunk.

          Show
          sandyr Sandy Ryza added a comment - Thanks for the review, Tom. Committed to branch-1, branch-2, branch-2.1-beta, and trunk.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #293 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/293/)
          MAPREDUCE-5367. Local jobs all use same local working directory (Sandy Ryza) (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1510610)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalJobRunner.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #293 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/293/ ) MAPREDUCE-5367 . Local jobs all use same local working directory (Sandy Ryza) (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1510610 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalJobRunner.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #1483 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1483/)
          MAPREDUCE-5367. Local jobs all use same local working directory (Sandy Ryza) (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1510610)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalJobRunner.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #1483 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1483/ ) MAPREDUCE-5367 . Local jobs all use same local working directory (Sandy Ryza) (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1510610 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalJobRunner.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #1510 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1510/)
          MAPREDUCE-5367. Local jobs all use same local working directory (Sandy Ryza) (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1510610)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalJobRunner.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1510 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1510/ ) MAPREDUCE-5367 . Local jobs all use same local working directory (Sandy Ryza) (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1510610 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalJobRunner.java
          Hide
          cnauroth Chris Nauroth added a comment -

          Hi, Sandy Ryza. Have you seen MAPREDUCE-5470? I'm wondering if we need to revert this patch for now, or if you think we can move ahead with another patch in MAPREDUCE-5470. Thanks!

          Show
          cnauroth Chris Nauroth added a comment - Hi, Sandy Ryza . Have you seen MAPREDUCE-5470 ? I'm wondering if we need to revert this patch for now, or if you think we can move ahead with another patch in MAPREDUCE-5470 . Thanks!

            People

            • Assignee:
              sandyr Sandy Ryza
              Reporter:
              sandyr Sandy Ryza
            • Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development