Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6909

LocalJobRunner fails when run on a node from multiple users

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.8.1
    • Fix Version/s: 2.9.0, 2.8.2
    • Component/s: client
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      MAPREDUCE-5762 removed mapreduce.jobtracker.staging.root.dir from mapred-default.xml but the property is still being used by LocalJobRunner and the code default value does not match the value that was removed from mapred-default.xml. This broke the use case where multiple users are running local mode jobs on the same node, since they now default to the same directory in /tmp.

        Issue Links

          Activity

          Hide
          jlowe Jason Lowe added a comment -

          Because the code default value didn't match the value in mapred-default.xml, removing it while it was still being used became an incompatible code change. Here's an example stack trace on 2.8 from a user running a local-mode job when some other user has already done so:

          ERROR 6017: JobID: job_local2035229526_0001 Reason: EACCES: Permission denied
                  at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native Method)
                  at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:230)
                  at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:767)
                  at org.apache.hadoop.fs.ChecksumFileSystem$1.apply(ChecksumFileSystem.java:506)
                  at org.apache.hadoop.fs.ChecksumFileSystem$FsOperation.run(ChecksumFileSystem.java:487)
                  at org.apache.hadoop.fs.ChecksumFileSystem.setPermission(ChecksumFileSystem.java:509)
                  at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:619)
                  at org.apache.hadoop.mapreduce.JobResourceUploader.uploadFiles(JobResourceUploader.java:94)
                  at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:97)
                  at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:192)
                  at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341)
                  at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1338)
                  at java.security.AccessController.doPrivileged(Native Method)
                  at javax.security.auth.Subject.doAs(Subject.java:422)
                  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1936)
                  at org.apache.hadoop.mapreduce.Job.submit(Job.java:1338)
                  at org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:335)
                  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                  at java.lang.reflect.Method.invoke(Method.java:498)
                  at org.apache.pig.backend.hadoop23.PigJobControl.submit(PigJobControl.java:128)
                  at org.apache.pig.backend.hadoop23.PigJobControl.run(PigJobControl.java:205)
                  at java.lang.Thread.run(Thread.java:745)
                  at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:288)
          
          Show
          jlowe Jason Lowe added a comment - Because the code default value didn't match the value in mapred-default.xml, removing it while it was still being used became an incompatible code change. Here's an example stack trace on 2.8 from a user running a local-mode job when some other user has already done so: ERROR 6017: JobID: job_local2035229526_0001 Reason: EACCES: Permission denied at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native Method) at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:230) at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:767) at org.apache.hadoop.fs.ChecksumFileSystem$1.apply(ChecksumFileSystem.java:506) at org.apache.hadoop.fs.ChecksumFileSystem$FsOperation.run(ChecksumFileSystem.java:487) at org.apache.hadoop.fs.ChecksumFileSystem.setPermission(ChecksumFileSystem.java:509) at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:619) at org.apache.hadoop.mapreduce.JobResourceUploader.uploadFiles(JobResourceUploader.java:94) at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:97) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:192) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1338) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1936) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1338) at org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:335) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.pig.backend.hadoop23.PigJobControl.submit(PigJobControl.java:128) at org.apache.pig.backend.hadoop23.PigJobControl.run(PigJobControl.java:205) at java.lang.Thread.run(Thread.java:745) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:288)
          Hide
          rchiang Ray Chiang added a comment -

          Could this be a duplicate of MAPREDUCE-6441?

          Show
          rchiang Ray Chiang added a comment - Could this be a duplicate of MAPREDUCE-6441 ?
          Hide
          jlowe Jason Lowe added a comment -

          No, MAPREDUCE-6441 is fixing a slightly different problem where multiple jobs are running from the same user. This is addressing a problem when two different users on the same node try to run local jobs, not necessarily at the same time. Even after the latest patch for MAPREDUCE-6441 is applied LocalJobRunner is still referencing JTConfig.JT_STAGING_AREA_ROOT, and that will break for multiple users after MAPREDUCE-5762 removed the default value for that property in mapred-default.xml.

          I went through the list of properties removed by MAPREDUCE-5762, and I created a patch that adds back any property that is being used by the code where the value in mapred-default.xml didn't match the default values in the code. One exception was mapreduce.cluster.temp.dir, but I suspect users are still referencing that value either directly or in their code.

          Show
          jlowe Jason Lowe added a comment - No, MAPREDUCE-6441 is fixing a slightly different problem where multiple jobs are running from the same user. This is addressing a problem when two different users on the same node try to run local jobs, not necessarily at the same time. Even after the latest patch for MAPREDUCE-6441 is applied LocalJobRunner is still referencing JTConfig.JT_STAGING_AREA_ROOT, and that will break for multiple users after MAPREDUCE-5762 removed the default value for that property in mapred-default.xml. I went through the list of properties removed by MAPREDUCE-5762 , and I created a patch that adds back any property that is being used by the code where the value in mapred-default.xml didn't match the default values in the code. One exception was mapreduce.cluster.temp.dir, but I suspect users are still referencing that value either directly or in their code.
          Hide
          jlowe Jason Lowe added a comment -

          This patch should be applied to branch-2 and branch-2.8. I'm assuming we're playing the "it's a major release so we are allowed to break stuff" card for the 3.x releases, so this doesn't need to go into trunk.

          Show
          jlowe Jason Lowe added a comment - This patch should be applied to branch-2 and branch-2.8. I'm assuming we're playing the "it's a major release so we are allowed to break stuff" card for the 3.x releases, so this doesn't need to go into trunk.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 21s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 6m 37s branch-2 passed
          +1 compile 0m 21s branch-2 passed with JDK v1.8.0_131
          +1 compile 0m 25s branch-2 passed with JDK v1.7.0_131
          +1 mvnsite 0m 31s branch-2 passed
          +1 mvneclipse 0m 16s branch-2 passed
          +1 javadoc 0m 21s branch-2 passed with JDK v1.8.0_131
          +1 javadoc 0m 25s branch-2 passed with JDK v1.7.0_131
          +1 mvninstall 0m 23s the patch passed
          +1 compile 0m 17s the patch passed with JDK v1.8.0_131
          +1 javac 0m 17s the patch passed
          +1 compile 0m 22s the patch passed with JDK v1.7.0_131
          +1 javac 0m 22s the patch passed
          +1 mvnsite 0m 29s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 javadoc 0m 17s the patch passed with JDK v1.8.0_131
          +1 javadoc 0m 21s the patch passed with JDK v1.7.0_131
          +1 unit 2m 35s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_131.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          17m 41s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:5e40efe
          JIRA Issue MAPREDUCE-6909
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12875280/MAPREDUCE-6909-branch-2.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
          uname Linux cea20173e8c7 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / 94bc5cd
          Default Java 1.7.0_131
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131
          JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7013/testReport/
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7013/console
          Powered by Apache Yetus 0.4.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 37s branch-2 passed +1 compile 0m 21s branch-2 passed with JDK v1.8.0_131 +1 compile 0m 25s branch-2 passed with JDK v1.7.0_131 +1 mvnsite 0m 31s branch-2 passed +1 mvneclipse 0m 16s branch-2 passed +1 javadoc 0m 21s branch-2 passed with JDK v1.8.0_131 +1 javadoc 0m 25s branch-2 passed with JDK v1.7.0_131 +1 mvninstall 0m 23s the patch passed +1 compile 0m 17s the patch passed with JDK v1.8.0_131 +1 javac 0m 17s the patch passed +1 compile 0m 22s the patch passed with JDK v1.7.0_131 +1 javac 0m 22s the patch passed +1 mvnsite 0m 29s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 javadoc 0m 17s the patch passed with JDK v1.8.0_131 +1 javadoc 0m 21s the patch passed with JDK v1.7.0_131 +1 unit 2m 35s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_131. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 17m 41s Subsystem Report/Notes Docker Image:yetus/hadoop:5e40efe JIRA Issue MAPREDUCE-6909 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12875280/MAPREDUCE-6909-branch-2.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux cea20173e8c7 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 94bc5cd Default Java 1.7.0_131 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131 JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7013/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7013/console Powered by Apache Yetus 0.4.0 http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 26m 42s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 8m 5s branch-2 passed
          +1 compile 0m 23s branch-2 passed with JDK v1.8.0_131
          +1 compile 0m 26s branch-2 passed with JDK v1.7.0_131
          +1 mvnsite 0m 32s branch-2 passed
          +1 mvneclipse 0m 16s branch-2 passed
          +1 javadoc 0m 21s branch-2 passed with JDK v1.8.0_131
          +1 javadoc 0m 27s branch-2 passed with JDK v1.7.0_131
          +1 mvninstall 0m 24s the patch passed
          +1 compile 0m 20s the patch passed with JDK v1.8.0_131
          +1 javac 0m 20s the patch passed
          +1 compile 0m 23s the patch passed with JDK v1.7.0_131
          +1 javac 0m 23s the patch passed
          +1 mvnsite 0m 30s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 0s The patch has no ill-formed XML file.
          +1 javadoc 0m 19s the patch passed with JDK v1.8.0_131
          +1 javadoc 0m 24s the patch passed with JDK v1.7.0_131
          +1 unit 2m 30s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_131.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          45m 49s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:5e40efe
          JIRA Issue MAPREDUCE-6909
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12875280/MAPREDUCE-6909-branch-2.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
          uname Linux fcd51d6ce2a3 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / 87c81cf
          Default Java 1.7.0_131
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131
          JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7014/testReport/
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7014/console
          Powered by Apache Yetus 0.4.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 26m 42s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 8m 5s branch-2 passed +1 compile 0m 23s branch-2 passed with JDK v1.8.0_131 +1 compile 0m 26s branch-2 passed with JDK v1.7.0_131 +1 mvnsite 0m 32s branch-2 passed +1 mvneclipse 0m 16s branch-2 passed +1 javadoc 0m 21s branch-2 passed with JDK v1.8.0_131 +1 javadoc 0m 27s branch-2 passed with JDK v1.7.0_131 +1 mvninstall 0m 24s the patch passed +1 compile 0m 20s the patch passed with JDK v1.8.0_131 +1 javac 0m 20s the patch passed +1 compile 0m 23s the patch passed with JDK v1.7.0_131 +1 javac 0m 23s the patch passed +1 mvnsite 0m 30s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. +1 javadoc 0m 19s the patch passed with JDK v1.8.0_131 +1 javadoc 0m 24s the patch passed with JDK v1.7.0_131 +1 unit 2m 30s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_131. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 45m 49s Subsystem Report/Notes Docker Image:yetus/hadoop:5e40efe JIRA Issue MAPREDUCE-6909 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12875280/MAPREDUCE-6909-branch-2.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux fcd51d6ce2a3 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 87c81cf Default Java 1.7.0_131 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131 JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7014/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7014/console Powered by Apache Yetus 0.4.0 http://yetus.apache.org This message was automatically generated.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          LGTM, +1. Thanks Jason for the fix.

          Show
          ajisakaa Akira Ajisaka added a comment - LGTM, +1. Thanks Jason for the fix.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Committed this to branch-2, branch-2.8, and branch-2.8.2. Thanks Jason Lowe for the fix!

          Show
          ajisakaa Akira Ajisaka added a comment - Committed this to branch-2, branch-2.8, and branch-2.8.2. Thanks Jason Lowe for the fix!

            People

            • Assignee:
              jlowe Jason Lowe
              Reporter:
              jlowe Jason Lowe
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development