Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3056

Jobs are failing when those are submitted by other users

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.23.0, 0.24.0
    • Fix Version/s: 0.23.0, 0.24.0
    • Component/s: applicationmaster, mrv2
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      MR cluster is started by the user 'root'. If any other users other than 'root' submit a job, it is failing always.

      Find the conatiner logs in the comments section.

      1. MAPREDUCE-3056.patch
        4 kB
        Devaraj K
      2. MAPREDUCE-3056-1.patch
        6 kB
        Devaraj K
      3. MAPREDUCE-3056-2.patch
        6 kB
        Devaraj K

        Activity

        Hide
        Devaraj K added a comment -
        2011-09-20 10:28:55,661 INFO  impl.JobImpl (JobImpl.java:setup(1001)) - DEBUG --- startJobs: parent=/tmp/hadoop-yarn/root/staging/root/.staging child=job_1316481926401_0003
        2011-09-20 10:28:55,667 INFO  impl.JobImpl (JobImpl.java:setup(1022)) - Adding job token for job_1316481926401_0003 to jobTokenSecretManager
        2011-09-20 10:28:55,987 INFO  impl.JobImpl (JobImpl.java:setup(1038)) - Writing back the job-token file on the remote file system:hdfs://10.18.52.124:9000/tmp/hadoop-yarn/root/staging/root/.staging/job_1316481926401_0003/appTokens
        2011-09-20 10:28:56,486 FATAL app.MRAppMaster (MRAppMaster.java:main(668)) - Error starting MRAppMaster
        org.apache.hadoop.yarn.YarnException: java.io.FileNotFoundException: File does not exist: hdfs://10.18.52.124:9000/tmp/hadoop-yarn/root/staging/root/.staging/job_1316481926401_0003/job.splitmetainfo
        	at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.createSplits(JobImpl.java:1113)
        	at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transition(JobImpl.java:838)
        	at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transition(JobImpl.java:809)
        	at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:376)
        	at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:294)
        	at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:39)
        	at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:439)
        	at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:684)
        	at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:117)
        	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:609)
        	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.start(MRAppMaster.java:563)
        	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:666)
        Caused by: java.io.FileNotFoundException: File does not exist: hdfs://10.18.52.124:9000/tmp/hadoop-yarn/root/staging/root/.staging/job_1316481926401_0003/job.splitmetainfo
        	at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:764)
        	at org.apache.hadoop.mapreduce.split.SplitMetaInfoReader.readSplitMetaInfo(SplitMetaInfoReader.java:50)
        	at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.createSplits(JobImpl.java:1108)
        	... 11 more
        2011-09-20 10:28:56,488 INFO  jobhistory.JobHistoryEventHandler (JobHistoryEventHandler.java:stop(238)) - Stopping JobHistoryEventHandler
        2011-09-20 10:28:56,488 INFO  service.CompositeService (CompositeService.java:stop(97)) - Error stopping JobHistoryEventHandler
        java.lang.NullPointerException
        	at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.stop(JobHistoryEventHandler.java:242)
        	at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:95)
        	at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:85)
        	at org.apache.hadoop.yarn.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:118)
        2011-09-20 10:28:56,489 INFO  service.CompositeService (CompositeService.java:stop(97)) - Error stopping org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter
        java.lang.NullPointerException
        	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter.stop(MRAppMaster.java:487)
        	at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:95)
        	at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:85)
        	at org.apache.hadoop.yarn.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:118)
        2011-09-20 10:28:56,489 INFO  service.CompositeService (CompositeService.java:stop(97)) - Error stopping org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter
        java.lang.NullPointerException
        	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.stop(MRAppMaster.java:443)
        	at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:95)
        	at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:85)
        	at org.apache.hadoop.yarn.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:118)
        2011-09-20 10:28:56,489 INFO  service.CompositeService (CompositeService.java:stop(97)) - Error stopping org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator
        java.lang.IllegalStateException: For this operation, current State must be STARTED instead of INITED
        	at org.apache.hadoop.yarn.service.AbstractService.ensureCurrentState(AbstractService.java:101)
        	at org.apache.hadoop.yarn.service.AbstractService.stop(AbstractService.java:69)
        	at org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator.stop(DefaultSpeculator.java:213)
        	at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:95)
        	at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:85)
        	at org.apache.hadoop.yarn.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:118)
        2011-09-20 10:28:56,491 INFO  service.CompositeService (CompositeService.java:stop(97)) - Error stopping MRClientService
        java.lang.NullPointerException
        	at org.apache.hadoop.mapreduce.v2.app.client.MRClientService.stop(MRClientService.java:161)
        	at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:95)
        	at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:85)
        	at org.apache.hadoop.yarn.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:118)
        2011-09-20 10:28:56,492 INFO  service.CompositeService (CompositeService.java:stop(97)) - Error stopping TaskCleaner
        java.lang.NullPointerException
        	at org.apache.hadoop.mapreduce.v2.app.taskclean.TaskCleanerImpl.stop(TaskCleanerImpl.java:73)
        	at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:95)
        	at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:85)
        	at org.apache.hadoop.yarn.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:118)
        2011-09-20 10:28:56,492 INFO  service.CompositeService (CompositeService.java:stop(97)) - Error stopping org.apache.hadoop.mapred.TaskAttemptListenerImpl
        java.lang.NullPointerException
        	at org.apache.hadoop.mapred.TaskAttemptListenerImpl.stopRpcServer(TaskAttemptListenerImpl.java:128)
        	at org.apache.hadoop.mapred.TaskAttemptListenerImpl.stop(TaskAttemptListenerImpl.java:123)
        	at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:95)
        	at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:85)
        	at org.apache.hadoop.yarn.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:118)
        2011-09-20 10:28:56,492 INFO  service.CompositeService (CompositeService.java:stop(97)) - Error stopping Dispatcher
        java.lang.NullPointerException
        	at org.apache.hadoop.yarn.event.AsyncDispatcher.stop(AsyncDispatcher.java:93)
        	at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:95)
        	at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:85)
        	at org.apache.hadoop.yarn.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:118)
        2011-09-20 10:28:56,493 INFO  service.CompositeService (CompositeService.java:run(120)) - Error stopping org.apache.hadoop.mapreduce.v2.app.MRAppMaster
        java.lang.IllegalStateException: For this operation, current State must be STARTED instead of INITED
        	at org.apache.hadoop.yarn.service.AbstractService.ensureCurrentState(AbstractService.java:101)
        	at org.apache.hadoop.yarn.service.AbstractService.stop(AbstractService.java:69)
        	at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:87)
        	at org.apache.hadoop.yarn.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:118)
        
        Show
        Devaraj K added a comment - 2011-09-20 10:28:55,661 INFO impl.JobImpl (JobImpl.java:setup(1001)) - DEBUG --- startJobs: parent=/tmp/hadoop-yarn/root/staging/root/.staging child=job_1316481926401_0003 2011-09-20 10:28:55,667 INFO impl.JobImpl (JobImpl.java:setup(1022)) - Adding job token for job_1316481926401_0003 to jobTokenSecretManager 2011-09-20 10:28:55,987 INFO impl.JobImpl (JobImpl.java:setup(1038)) - Writing back the job-token file on the remote file system:hdfs://10.18.52.124:9000/tmp/hadoop-yarn/root/staging/root/.staging/job_1316481926401_0003/appTokens 2011-09-20 10:28:56,486 FATAL app.MRAppMaster (MRAppMaster.java:main(668)) - Error starting MRAppMaster org.apache.hadoop.yarn.YarnException: java.io.FileNotFoundException: File does not exist: hdfs://10.18.52.124:9000/tmp/hadoop-yarn/root/staging/root/.staging/job_1316481926401_0003/job.splitmetainfo at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.createSplits(JobImpl.java:1113) at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transition(JobImpl.java:838) at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transition(JobImpl.java:809) at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:376) at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:294) at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:39) at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:439) at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:684) at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:117) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:609) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.start(MRAppMaster.java:563) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:666) Caused by: java.io.FileNotFoundException: File does not exist: hdfs://10.18.52.124:9000/tmp/hadoop-yarn/root/staging/root/.staging/job_1316481926401_0003/job.splitmetainfo at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:764) at org.apache.hadoop.mapreduce.split.SplitMetaInfoReader.readSplitMetaInfo(SplitMetaInfoReader.java:50) at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.createSplits(JobImpl.java:1108) ... 11 more 2011-09-20 10:28:56,488 INFO jobhistory.JobHistoryEventHandler (JobHistoryEventHandler.java:stop(238)) - Stopping JobHistoryEventHandler 2011-09-20 10:28:56,488 INFO service.CompositeService (CompositeService.java:stop(97)) - Error stopping JobHistoryEventHandler java.lang.NullPointerException at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.stop(JobHistoryEventHandler.java:242) at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:95) at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:85) at org.apache.hadoop.yarn.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:118) 2011-09-20 10:28:56,489 INFO service.CompositeService (CompositeService.java:stop(97)) - Error stopping org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter java.lang.NullPointerException at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter.stop(MRAppMaster.java:487) at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:95) at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:85) at org.apache.hadoop.yarn.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:118) 2011-09-20 10:28:56,489 INFO service.CompositeService (CompositeService.java:stop(97)) - Error stopping org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter java.lang.NullPointerException at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.stop(MRAppMaster.java:443) at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:95) at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:85) at org.apache.hadoop.yarn.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:118) 2011-09-20 10:28:56,489 INFO service.CompositeService (CompositeService.java:stop(97)) - Error stopping org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator java.lang.IllegalStateException: For this operation, current State must be STARTED instead of INITED at org.apache.hadoop.yarn.service.AbstractService.ensureCurrentState(AbstractService.java:101) at org.apache.hadoop.yarn.service.AbstractService.stop(AbstractService.java:69) at org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator.stop(DefaultSpeculator.java:213) at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:95) at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:85) at org.apache.hadoop.yarn.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:118) 2011-09-20 10:28:56,491 INFO service.CompositeService (CompositeService.java:stop(97)) - Error stopping MRClientService java.lang.NullPointerException at org.apache.hadoop.mapreduce.v2.app.client.MRClientService.stop(MRClientService.java:161) at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:95) at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:85) at org.apache.hadoop.yarn.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:118) 2011-09-20 10:28:56,492 INFO service.CompositeService (CompositeService.java:stop(97)) - Error stopping TaskCleaner java.lang.NullPointerException at org.apache.hadoop.mapreduce.v2.app.taskclean.TaskCleanerImpl.stop(TaskCleanerImpl.java:73) at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:95) at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:85) at org.apache.hadoop.yarn.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:118) 2011-09-20 10:28:56,492 INFO service.CompositeService (CompositeService.java:stop(97)) - Error stopping org.apache.hadoop.mapred.TaskAttemptListenerImpl java.lang.NullPointerException at org.apache.hadoop.mapred.TaskAttemptListenerImpl.stopRpcServer(TaskAttemptListenerImpl.java:128) at org.apache.hadoop.mapred.TaskAttemptListenerImpl.stop(TaskAttemptListenerImpl.java:123) at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:95) at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:85) at org.apache.hadoop.yarn.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:118) 2011-09-20 10:28:56,492 INFO service.CompositeService (CompositeService.java:stop(97)) - Error stopping Dispatcher java.lang.NullPointerException at org.apache.hadoop.yarn.event.AsyncDispatcher.stop(AsyncDispatcher.java:93) at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:95) at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:85) at org.apache.hadoop.yarn.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:118) 2011-09-20 10:28:56,493 INFO service.CompositeService (CompositeService.java:run(120)) - Error stopping org.apache.hadoop.mapreduce.v2.app.MRAppMaster java.lang.IllegalStateException: For this operation, current State must be STARTED instead of INITED at org.apache.hadoop.yarn.service.AbstractService.ensureCurrentState(AbstractService.java:101) at org.apache.hadoop.yarn.service.AbstractService.stop(AbstractService.java:69) at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:87) at org.apache.hadoop.yarn.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:118)
        Hide
        Mahadev konar added a comment -

        @Devaraj,
        Looks like the staging dir still is pointing to roots dir?

        java.io.FileNotFoundException: File does not exist: hdfs://10.18.52.124:9000/tmp/hadoop-yarn/root/staging/root/.staging/job_1316481926401_0003/job.splitmetainfo

        Do you have the client trace? What staging dir is shown on the client?

        Show
        Mahadev konar added a comment - @Devaraj, Looks like the staging dir still is pointing to roots dir? java.io.FileNotFoundException: File does not exist: hdfs://10.18.52.124:9000/tmp/hadoop-yarn/root/staging/root/.staging/job_1316481926401_0003/job.splitmetainfo Do you have the client trace? What staging dir is shown on the client?
        Hide
        Devaraj K added a comment -

        Yes Mahadev. App Master is still referring the root's staging dir.

        Client is showing the staging dir as 'hdfs://10.18.52.124:9000/tmp/hadoop-yarn/devaraj/staging/devaraj/.staging' when the submitted user name is 'devaraj'.

        Show
        Devaraj K added a comment - Yes Mahadev. App Master is still referring the root's staging dir. Client is showing the staging dir as 'hdfs://10.18.52.124:9000/tmp/hadoop-yarn/devaraj/staging/devaraj/.staging' when the submitted user name is 'devaraj'.
        Hide
        Vinod Kumar Vavilapalli added a comment -

        I ran into this today myself. This happens with DefaultContainerExecutor case, which most of us are not using (surprise )

        Devaraj (K), can you confirm?

        The problem is the default staging dir use of $

        {user.name}

        system property, which is the job-submitter on the client side and the system user on the server side.

        Show
        Vinod Kumar Vavilapalli added a comment - I ran into this today myself. This happens with DefaultContainerExecutor case, which most of us are not using (surprise ) Devaraj (K), can you confirm? The problem is the default staging dir use of $ {user.name} system property, which is the job-submitter on the client side and the system user on the server side.
        Hide
        Aaron T. Myers added a comment -

        FWIW, basically the same bug exists in 0.20.x when $

        {user.name} is used in the value of the mapred.local.dir with the LinuxTaskController. I'd personally like to completely abandon support for expanding ${user.name}

        in configs for exactly this reason (HADOOP-7522) but in the absence of that, removing it from default values is certainly a step in the right direction.

        Show
        Aaron T. Myers added a comment - FWIW, basically the same bug exists in 0.20.x when $ {user.name} is used in the value of the mapred.local.dir with the LinuxTaskController . I'd personally like to completely abandon support for expanding ${user.name} in configs for exactly this reason ( HADOOP-7522 ) but in the absence of that, removing it from default values is certainly a step in the right direction.
        Hide
        Devaraj K added a comment -

        1. The default value of staging dir uses the $

        {user.name}

        . It is getting substituted in the app master with the server user name but it should use the client user name.

        For this issue, as per ATM suggestion I have removed the user.name from the default staging dir.

        2. Here the staging area dir uses the user name. It was using the system user name for this in couple of places and I have changed it to job submitted user name.

        MRApps.java
         
          public static Path getStagingAreaDir(Configuration conf, String user) { 
            return new Path( 
                conf.get(MRJobConfig.MR_AM_STAGING_DIR) + 
                Path.SEPARATOR + user + Path.SEPARATOR + STAGING_CONSTANT); 
          } 
        
        
        Show
        Devaraj K added a comment - 1. The default value of staging dir uses the $ {user.name} . It is getting substituted in the app master with the server user name but it should use the client user name. For this issue, as per ATM suggestion I have removed the user.name from the default staging dir. 2. Here the staging area dir uses the user name. It was using the system user name for this in couple of places and I have changed it to job submitted user name. MRApps.java public static Path getStagingAreaDir(Configuration conf, String user) { return new Path( conf.get(MRJobConfig.MR_AM_STAGING_DIR) + Path.SEPARATOR + user + Path.SEPARATOR + STAGING_CONSTANT); }
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12497139/MAPREDUCE-3056.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/906//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/906//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12497139/MAPREDUCE-3056.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/906//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/906//console This message is automatically generated.
        Hide
        Devaraj K added a comment -

        I have verified it manually, jobs submitted by other users are executing fine. Please review this.

        Show
        Devaraj K added a comment - I have verified it manually, jobs submitted by other users are executing fine. Please review this.
        Hide
        Aaron T. Myers added a comment -

        Hey Devaraj, it should be possible to write a test for this, right? Or am I missing something that would make doing so difficult?

        Show
        Aaron T. Myers added a comment - Hey Devaraj, it should be possible to write a test for this, right? Or am I missing something that would make doing so difficult?
        Hide
        Arun C Murthy added a comment -

        Agree with atm. Please provide a test case. Thanks!

        Show
        Arun C Murthy added a comment - Agree with atm. Please provide a test case. Thanks!
        Hide
        Arun C Murthy added a comment -

        Also, don't get user name for config. Always use UGI.getCurrentUser.

        Show
        Arun C Murthy added a comment - Also, don't get user name for config. Always use UGI.getCurrentUser.
        Hide
        Devaraj K added a comment -

        Thanks for looking into the patch.

        I have updated the patch as per above comments.

        Show
        Devaraj K added a comment - Thanks for looking into the patch. I have updated the patch as per above comments.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12497470/MAPREDUCE-3056-1.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/926//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/926//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12497470/MAPREDUCE-3056-1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/926//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/926//console This message is automatically generated.
        Hide
        Vinod Kumar Vavilapalli added a comment -

        You missed changes for mapred-default.xml in the latest patch.

        MRAppMaster:

        • Wrap lines >80 chars
        • MRAppMaster is a process started by NM and so current-user has no tokens loaded. You can drop the loading of tokens from current-user into the created user.
        • You could collapse MRAppMaster.initAndStartAppMaster() inline and directly invoke main() in the test mocking the System.getenv() calls. Please try if it is possible, otherwise I am okay if you drop this.

        Thanks!

        Show
        Vinod Kumar Vavilapalli added a comment - You missed changes for mapred-default.xml in the latest patch. MRAppMaster: Wrap lines >80 chars MRAppMaster is a process started by NM and so current-user has no tokens loaded. You can drop the loading of tokens from current-user into the created user. You could collapse MRAppMaster.initAndStartAppMaster() inline and directly invoke main() in the test mocking the System.getenv() calls. Please try if it is possible, otherwise I am okay if you drop this. Thanks!
        Hide
        Devaraj K added a comment -

        Thanks Vinod.

        You missed changes for mapred-default.xml in the latest patch.

        I,m sorry, missed this. I have included in the latest patch.

        Wrap lines >80 chars

        I have taken care in the updated patch.

        MRAppMaster is a process started by NM and so current-user has no tokens loaded. You can drop the loading of tokens from current-user into the created user.

        Removed the loading of tokens.

        You could collapse MRAppMaster.initAndStartAppMaster() inline and directly invoke main() in the test mocking the System.getenv() calls. Please try if it is possible, otherwise I am okay if you drop this.

        I have not done this change because it requires for the testcase.

        Show
        Devaraj K added a comment - Thanks Vinod. You missed changes for mapred-default.xml in the latest patch. I,m sorry, missed this. I have included in the latest patch. Wrap lines >80 chars I have taken care in the updated patch. MRAppMaster is a process started by NM and so current-user has no tokens loaded. You can drop the loading of tokens from current-user into the created user. Removed the loading of tokens. You could collapse MRAppMaster.initAndStartAppMaster() inline and directly invoke main() in the test mocking the System.getenv() calls. Please try if it is possible, otherwise I am okay if you drop this. I have not done this change because it requires for the testcase.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12497591/MAPREDUCE-3056-2.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/933//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/933//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12497591/MAPREDUCE-3056-2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/933//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/933//console This message is automatically generated.
        Hide
        Vinod Kumar Vavilapalli added a comment -

        +1.

        I just committed this to trunk and branch-0.23. Thanks Devaraj (K)!

        Show
        Vinod Kumar Vavilapalli added a comment - +1. I just committed this to trunk and branch-0.23. Thanks Devaraj (K)!
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #1029 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1029/)
        MAPREDUCE-3056. Changed the default staging directory to not include user.name to prevent issues with non-secure mode. Contributed by Devaraj K.

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178740
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1029 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1029/ ) MAPREDUCE-3056 . Changed the default staging directory to not include user.name to prevent issues with non-secure mode. Contributed by Devaraj K. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178740 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #1009 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1009/)
        MAPREDUCE-3056. Changed the default staging directory to not include user.name to prevent issues with non-secure mode. Contributed by Devaraj K.

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178740
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1009 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1009/ ) MAPREDUCE-3056 . Changed the default staging directory to not include user.name to prevent issues with non-secure mode. Contributed by Devaraj K. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178740 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #1087 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1087/)
        MAPREDUCE-3056. Changed the default staging directory to not include user.name to prevent issues with non-secure mode. Contributed by Devaraj K.

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178740
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1087 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1087/ ) MAPREDUCE-3056 . Changed the default staging directory to not include user.name to prevent issues with non-secure mode. Contributed by Devaraj K. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178740 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Build #36 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/36/)
        MAPREDUCE-3056. svn merge -c r1178740 --ignore-ancestry ../../trunk/

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178741
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #36 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/36/ ) MAPREDUCE-3056 . svn merge -c r1178740 --ignore-ancestry ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178741 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #820 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/820/)
        MAPREDUCE-3056. Changed the default staging directory to not include user.name to prevent issues with non-secure mode. Contributed by Devaraj K.

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178740
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #820 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/820/ ) MAPREDUCE-3056 . Changed the default staging directory to not include user.name to prevent issues with non-secure mode. Contributed by Devaraj K. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178740 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #29 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/29/)
        MAPREDUCE-3056. svn merge -c r1178740 --ignore-ancestry ../../trunk/

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178741
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #29 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/29/ ) MAPREDUCE-3056 . svn merge -c r1178740 --ignore-ancestry ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178741 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #850 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/850/)
        MAPREDUCE-3056. Changed the default staging directory to not include user.name to prevent issues with non-secure mode. Contributed by Devaraj K.

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178740
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #850 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/850/ ) MAPREDUCE-3056 . Changed the default staging directory to not include user.name to prevent issues with non-secure mode. Contributed by Devaraj K. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178740 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml

          People

          • Assignee:
            Devaraj K
            Reporter:
            Devaraj K
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development