Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3056

Jobs are failing when those are submitted by other users

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.23.0, 2.0.0-alpha
    • Fix Version/s: 0.23.0
    • Component/s: applicationmaster, mrv2
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      MR cluster is started by the user 'root'. If any other users other than 'root' submit a job, it is failing always.

      Find the conatiner logs in the comments section.

      1. MAPREDUCE-3056.patch
        4 kB
        Devaraj K
      2. MAPREDUCE-3056-1.patch
        6 kB
        Devaraj K
      3. MAPREDUCE-3056-2.patch
        6 kB
        Devaraj K

        Activity

        Hide
        Devaraj K added a comment -

        Sergey, this jira was fixed long back. If you have any queries please use the mailing lists. If you feel this is a bug, create an another jira with the details. Thanks.

        Show
        Devaraj K added a comment - Sergey , this jira was fixed long back. If you have any queries please use the mailing lists. If you feel this is a bug, create an another jira with the details. Thanks.
        Hide
        Sergey added a comment -

        Hi, not sure, but I hit described problem:
        CDH-5.3.2-1.cdh5.3.2.p0.10

        ```
        2015-05-06 18:56:35,606 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1430916274046_0418_000001
        2015-05-06 18:56:35,905 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: hadoop.ssl.require.client.cert; Ignoring.
        2015-05-06 18:56:35,908 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
        2015-05-06 18:56:35,909 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: hadoop.ssl.client.conf; Ignoring.
        2015-05-06 18:56:35,911 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: hadoop.ssl.keystores.factory.class; Ignoring.
        2015-05-06 18:56:35,915 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: hadoop.ssl.server.conf; Ignoring.
        2015-05-06 18:56:35,932 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
        2015-05-06 18:56:36,033 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
        2015-05-06 18:56:36,033 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN, Service: , Ident: (org.apache.hadoop.yarn.security.AMRMTokenIdentifier@1c667739)
        2015-05-06 18:56:36,068 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: RM_DELEGATION_TOKEN, Service: 10.66.62.141:8032,10.66.62.146:8032, Ident: (owner=devops, renewer=oozie mr token, realUser=oozie, issueDate=1430927768380, maxDate=1431532568380, sequenceNumber=70067, masterKeyId=110)
        2015-05-06 18:56:36,083 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Using mapred newApiCommitter.
        2015-05-06 18:56:36,181 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: hadoop.ssl.require.client.cert; Ignoring.
        2015-05-06 18:56:36,183 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
        2015-05-06 18:56:36,183 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: hadoop.ssl.client.conf; Ignoring.
        2015-05-06 18:56:36,184 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: hadoop.ssl.keystores.factory.class; Ignoring.
        2015-05-06 18:56:36,187 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: hadoop.ssl.server.conf; Ignoring.
        2015-05-06 18:56:36,197 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
        2015-05-06 18:56:36,925 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config null
        2015-05-06 18:56:37,054 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter is com.linkedin.camus.etl.kafka.mapred.EtlMultiOutputCommitter
        2015-05-06 18:56:37,080 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.jobhistory.EventType for class org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler
        2015-05-06 18:56:37,081 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher
        2015-05-06 18:56:37,082 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher
        2015-05-06 18:56:37,083 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher
        2015-05-06 18:56:37,083 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler
        2015-05-06 18:56:37,089 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.speculate.Speculator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$SpeculatorEventDispatcher
        2015-05-06 18:56:37,090 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter
        2015-05-06 18:56:37,091 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncher$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter
        2015-05-06 18:56:37,107 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Perms after creating 448, Expected: 448
        2015-05-06 18:56:37,156 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler
        2015-05-06 18:56:37,444 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
        2015-05-06 18:56:37,504 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
        2015-05-06 18:56:37,504 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MRAppMaster metrics system started
        2015-05-06 18:56:37,513 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Adding job token for job_1430916274046_0418 to jobTokenSecretManager
        2015-05-06 18:56:37,535 WARN [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Job init failed
        org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.FileNotFoundException: File does not exist: hdfs://nameservice1/user/devops/.staging/job_1430916274046_0418/job.splitmetainfo
        at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.createSplits(JobImpl.java:1566)
        at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transition(JobImpl.java:1430)
        at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transition(JobImpl.java:1388)
        at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
        at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
        at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
        at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
        at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:996)
        at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:138)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:1272)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1045)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1478)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1474)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1407)
        Caused by: java.io.FileNotFoundException: File does not exist: hdfs://nameservice1/user/devops/.staging/job_1430916274046_0418/job.splitmetainfo
        at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1093)
        at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1085)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1085)
        at org.apache.hadoop.mapreduce.split.SplitMetaInfoReader.readSplitMetaInfo(SplitMetaInfoReader.java:51)
        at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.createSplits(JobImpl.java:1561)
        ... 17 more
        2015-05-06 18:56:37,542 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster launching normal, non-uberized, multi-container job job_1430916274046_0418.
        2015-05-06 18:56:37,578 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue class java.util.concurrent.LinkedBlockingQueue
        2015-05-06 18:56:37,588 INFO Socket Reader #1 for port 52894 org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 52894
        2015-05-06 18:56:37,609 INFO [main] org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding protocol org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB to the server
        2015-05-06 18:56:37,609 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting
        2015-05-06 18:56:37,609 INFO [IPC Server listener on 52894] org.apache.hadoop.ipc.Server: IPC Server listener on 52894: starting
        2015-05-06 18:56:37,610 INFO [main] org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Instantiated MRClientService at prod-node0149.kyc.myhost.ru/10.66.62.122:52894
        2015-05-06 18:56:37,675 INFO [main] org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
        2015-05-06 18:56:37,679 INFO [main] org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.mapreduce is not defined
        2015-05-06 18:56:37,703 INFO [main] org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
        2015-05-06 18:56:37,767 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: hadoop.ssl.require.client.cert; Ignoring.
        2015-05-06 18:56:37,769 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
        2015-05-06 18:56:37,769 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: hadoop.ssl.client.conf; Ignoring.
        2015-05-06 18:56:37,770 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: hadoop.ssl.keystores.factory.class; Ignoring.
        2015-05-06 18:56:37,771 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: hadoop.ssl.server.conf; Ignoring.
        2015-05-06 18:56:37,779 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
        2015-05-06 18:56:37,784 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context mapreduce
        2015-05-06 18:56:37,784 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context static
        2015-05-06 18:56:37,788 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /mapreduce/*
        2015-05-06 18:56:37,788 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /ws/*
        2015-05-06 18:56:37,800 INFO [main] org.apache.hadoop.http.HttpServer2: Jetty bound to port 43170
        2015-05-06 18:56:37,800 INFO [main] org.mortbay.log: jetty-6.1.26.cloudera.4
        2015-05-06 18:56:37,827 INFO [main] org.mortbay.log: Extract jar:file:/opt/cloudera/parcels/CDH-5.3.2-1.cdh5.3.2.p0.10/jars/hadoop-yarn-common-2.5.0-cdh5.3.2.jar!/webapps/mapreduce to /tmp/Jetty_0_0_0_0_43170_mapreduce____.3fyrma/webapp
        2015-05-06 18:56:38,170 INFO [main] org.mortbay.log: Started HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:43170
        2015-05-06 18:56:38,170 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Web app /mapreduce started at 43170
        2015-05-06 18:56:38,496 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Registered webapp guice modules
        2015-05-06 18:56:38,499 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator: JOB_CREATE job_1430916274046_0418
        2015-05-06 18:56:38,501 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue class java.util.concurrent.LinkedBlockingQueue
        2015-05-06 18:56:38,501 INFO Socket Reader #1 for port 34861 org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 34861
        2015-05-06 18:56:38,506 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting
        2015-05-06 18:56:38,506 INFO [IPC Server listener on 34861] org.apache.hadoop.ipc.Server: IPC Server listener on 34861: starting
        2015-05-06 18:56:38,525 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: nodeBlacklistingEnabled:true
        2015-05-06 18:56:38,526 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: maxTaskFailuresPerNode is 3
        2015-05-06 18:56:38,526 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: blacklistDisablePercent is 33
        2015-05-06 18:56:38,596 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: hadoop.ssl.require.client.cert; Ignoring.
        2015-05-06 18:56:38,596 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
        2015-05-06 18:56:38,597 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: hadoop.ssl.client.conf; Ignoring.
        2015-05-06 18:56:38,597 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: hadoop.ssl.keystores.factory.class; Ignoring.
        2015-05-06 18:56:38,598 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: hadoop.ssl.server.conf; Ignoring.
        2015-05-06 18:56:38,602 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
        2015-05-06 18:56:38,663 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: maxContainerCapability: <memory:202752, vCores:34>
        2015-05-06 18:56:38,664 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: queue: root.masterdata
        2015-05-06 18:56:38,667 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Upper limit on the thread pool size is 500
        2015-05-06 18:56:38,669 INFO [main] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0
        2015-05-06 18:56:38,698 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1430916274046_0418Job Transitioned from NEW to FAIL_ABORT
        2015-05-06 18:56:38,700 INFO CommitterEvent Processor #0 org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_ABORT
        2015-05-06 18:56:38,709 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1430916274046_0418Job Transitioned from FAIL_ABORT to FAILED
        2015-05-06 18:56:38,709 INFO [Thread-55] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: We are finishing cleanly so this is the last retry
        2015-05-06 18:56:38,709 INFO [Thread-55] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify RMCommunicator isAMLastRetry: true
        2015-05-06 18:56:38,709 INFO [Thread-55] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: RMCommunicator notified that shouldUnregistered is: true
        2015-05-06 18:56:38,709 INFO [Thread-55] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify JHEH isAMLastRetry: true
        2015-05-06 18:56:38,709 INFO [Thread-55] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: JobHistoryEventHandler notified that forceJobCompletion is true
        2015-05-06 18:56:38,709 INFO [Thread-55] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Calling stop for all the services
        2015-05-06 18:56:38,710 INFO [Thread-55] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopping JobHistoryEventHandler. Size of the outstanding queue size is 3
        2015-05-06 18:56:38,807 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event Writer setup for JobId: job_1430916274046_0418, File: hdfs://nameservice1/user/devops/.staging/job_1430916274046_0418/job_1430916274046_0418_1.jhist
        2015-05-06 18:56:39,144 INFO [Thread-55] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: In stop, writing event JOB_SUBMITTED
        2015-05-06 18:56:39,157 INFO [Thread-55] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: In stop, writing event JOB_QUEUE_CHANGED
        2015-05-06 18:56:39,157 INFO [Thread-55] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: In stop, writing event JOB_FAILED
        2015-05-06 18:56:39,215 INFO [Thread-55] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying hdfs://nameservice1/user/devops/.staging/job_1430916274046_0418/job_1430916274046_0418_1.jhist to hdfs://nameservice1/user/history/done_intermediate/devops/job_1430916274046_0418-1430927787783-devops-Camus+job+%2D+cisgw%2Dmonitoring-1430927798706-0-0-FAILED-root.masterdata-1430927798706.jhist_tmp
        2015-05-06 18:56:39,299 INFO [Thread-55] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied to done location: hdfs://nameservice1/user/history/done_intermediate/devops/job_1430916274046_0418-1430927787783-devops-Camus+job+%2D+cisgw%2Dmonitoring-1430927798706-0-0-FAILED-root.masterdata-1430927798706.jhist_tmp
        2015-05-06 18:56:39,302 INFO [Thread-55] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying hdfs://nameservice1/user/devops/.staging/job_1430916274046_0418/job_1430916274046_0418_1_conf.xml to hdfs://nameservice1/user/history/done_intermediate/devops/job_1430916274046_0418_conf.xml_tmp
        2015-05-06 18:56:39,319 INFO [Thread-55] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied to done location: hdfs://nameservice1/user/history/done_intermediate/devops/job_1430916274046_0418_conf.xml_tmp
        2015-05-06 18:56:39,325 INFO [Thread-55] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://nameservice1/user/history/done_intermediate/devops/job_1430916274046_0418.summary_tmp to hdfs://nameservice1/user/history/done_intermediate/devops/job_1430916274046_0418.summary
        2015-05-06 18:56:39,327 INFO [Thread-55] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://nameservice1/user/history/done_intermediate/devops/job_1430916274046_0418_conf.xml_tmp to hdfs://nameservice1/user/history/done_intermediate/devops/job_1430916274046_0418_conf.xml
        2015-05-06 18:56:39,328 INFO [Thread-55] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://nameservice1/user/history/done_intermediate/devops/job_1430916274046_0418-1430927787783-devops-Camus+job+%2D+cisgw%2Dmonitoring-1430927798706-0-0-FAILED-root.masterdata-1430927798706.jhist_tmp to hdfs://nameservice1/user/history/done_intermediate/devops/job_1430916274046_0418-1430927787783-devops-Camus+job+%2D+cisgw%2Dmonitoring-1430927798706-0-0-FAILED-root.masterdata-1430927798706.jhist
        2015-05-06 18:56:39,328 INFO [Thread-55] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopped JobHistoryEventHandler. super.stop()
        2015-05-06 18:56:39,329 INFO [Thread-55] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Setting job diagnostics to Job init failed : org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.FileNotFoundException: File does not exist: hdfs://nameservice1/user/devops/.staging/job_1430916274046_0418/job.splitmetainfo
        at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.createSplits(JobImpl.java:1566)
        at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transition(JobImpl.java:1430)
        at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transition(JobImpl.java:1388)
        at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
        at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
        at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
        at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
        at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:996)
        at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:138)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:1272)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1045)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1478)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1474)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1407)
        Caused by: java.io.FileNotFoundException: File does not exist: hdfs://nameservice1/user/devops/.staging/job_1430916274046_0418/job.splitmetainfo
        at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1093)
        at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1085)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1085)
        at org.apache.hadoop.mapreduce.split.SplitMetaInfoReader.readSplitMetaInfo(SplitMetaInfoReader.java:51)
        at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.createSplits(JobImpl.java:1561)
        ... 17 more

        2015-05-06 18:56:39,331 INFO [Thread-55] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: History url is http://prod-node0138.kyc.myhost.ru:19888/jobhistory/job/job_1430916274046_0418
        2015-05-06 18:56:39,339 INFO [Thread-55] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for application to be successfully unregistered.
        2015-05-06 18:56:40,340 INFO [Thread-55] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Final Stats: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0 HostLocal:0 RackLocal:0
        2015-05-06 18:56:40,341 INFO [Thread-55] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Deleting staging directory hdfs://nameservice1 /user/yarn/.staging/job_1430916274046_0418
        2015-05-06 18:56:40,343 INFO [Thread-55] org.apache.hadoop.ipc.Server: Stopping server on 34861
        2015-05-06 18:56:40,343 INFO [IPC Server listener on 34861] org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 34861
        2015-05-06 18:56:40,346 INFO [TaskHeartbeatHandler PingChecker] org.apache.hadoop.mapreduce.v2.app.TaskHeartbeatHandler: TaskHeartbeatHandler thread interrupted
        2015-05-06 18:56:40,346 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: Stopping IPC Server Responder
        ```

        Show
        Sergey added a comment - Hi, not sure, but I hit described problem: CDH-5.3.2-1.cdh5.3.2.p0.10 ``` 2015-05-06 18:56:35,606 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1430916274046_0418_000001 2015-05-06 18:56:35,905 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: hadoop.ssl.require.client.cert; Ignoring. 2015-05-06 18:56:35,908 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring. 2015-05-06 18:56:35,909 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: hadoop.ssl.client.conf; Ignoring. 2015-05-06 18:56:35,911 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: hadoop.ssl.keystores.factory.class; Ignoring. 2015-05-06 18:56:35,915 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: hadoop.ssl.server.conf; Ignoring. 2015-05-06 18:56:35,932 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring. 2015-05-06 18:56:36,033 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens: 2015-05-06 18:56:36,033 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN, Service: , Ident: (org.apache.hadoop.yarn.security.AMRMTokenIdentifier@1c667739) 2015-05-06 18:56:36,068 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: RM_DELEGATION_TOKEN, Service: 10.66.62.141:8032,10.66.62.146:8032, Ident: (owner=devops, renewer=oozie mr token, realUser=oozie, issueDate=1430927768380, maxDate=1431532568380, sequenceNumber=70067, masterKeyId=110) 2015-05-06 18:56:36,083 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Using mapred newApiCommitter. 2015-05-06 18:56:36,181 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: hadoop.ssl.require.client.cert; Ignoring. 2015-05-06 18:56:36,183 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring. 2015-05-06 18:56:36,183 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: hadoop.ssl.client.conf; Ignoring. 2015-05-06 18:56:36,184 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: hadoop.ssl.keystores.factory.class; Ignoring. 2015-05-06 18:56:36,187 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: hadoop.ssl.server.conf; Ignoring. 2015-05-06 18:56:36,197 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring. 2015-05-06 18:56:36,925 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config null 2015-05-06 18:56:37,054 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter is com.linkedin.camus.etl.kafka.mapred.EtlMultiOutputCommitter 2015-05-06 18:56:37,080 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.jobhistory.EventType for class org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler 2015-05-06 18:56:37,081 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher 2015-05-06 18:56:37,082 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher 2015-05-06 18:56:37,083 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher 2015-05-06 18:56:37,083 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler 2015-05-06 18:56:37,089 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.speculate.Speculator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$SpeculatorEventDispatcher 2015-05-06 18:56:37,090 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter 2015-05-06 18:56:37,091 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncher$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter 2015-05-06 18:56:37,107 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Perms after creating 448, Expected: 448 2015-05-06 18:56:37,156 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler 2015-05-06 18:56:37,444 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties 2015-05-06 18:56:37,504 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2015-05-06 18:56:37,504 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MRAppMaster metrics system started 2015-05-06 18:56:37,513 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Adding job token for job_1430916274046_0418 to jobTokenSecretManager 2015-05-06 18:56:37,535 WARN [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Job init failed org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.FileNotFoundException: File does not exist: hdfs://nameservice1/user/devops/.staging/job_1430916274046_0418/job.splitmetainfo at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.createSplits(JobImpl.java:1566) at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transition(JobImpl.java:1430) at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transition(JobImpl.java:1388) at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385) at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302) at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46) at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448) at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:996) at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:138) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:1272) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1045) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1478) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1474) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1407) Caused by: java.io.FileNotFoundException: File does not exist: hdfs://nameservice1/user/devops/.staging/job_1430916274046_0418/job.splitmetainfo at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1093) at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1085) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1085) at org.apache.hadoop.mapreduce.split.SplitMetaInfoReader.readSplitMetaInfo(SplitMetaInfoReader.java:51) at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.createSplits(JobImpl.java:1561) ... 17 more 2015-05-06 18:56:37,542 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster launching normal, non-uberized, multi-container job job_1430916274046_0418. 2015-05-06 18:56:37,578 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue class java.util.concurrent.LinkedBlockingQueue 2015-05-06 18:56:37,588 INFO Socket Reader #1 for port 52894 org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 52894 2015-05-06 18:56:37,609 INFO [main] org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding protocol org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB to the server 2015-05-06 18:56:37,609 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting 2015-05-06 18:56:37,609 INFO [IPC Server listener on 52894] org.apache.hadoop.ipc.Server: IPC Server listener on 52894: starting 2015-05-06 18:56:37,610 INFO [main] org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Instantiated MRClientService at prod-node0149.kyc.myhost.ru/10.66.62.122:52894 2015-05-06 18:56:37,675 INFO [main] org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2015-05-06 18:56:37,679 INFO [main] org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.mapreduce is not defined 2015-05-06 18:56:37,703 INFO [main] org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter) 2015-05-06 18:56:37,767 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: hadoop.ssl.require.client.cert; Ignoring. 2015-05-06 18:56:37,769 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring. 2015-05-06 18:56:37,769 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: hadoop.ssl.client.conf; Ignoring. 2015-05-06 18:56:37,770 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: hadoop.ssl.keystores.factory.class; Ignoring. 2015-05-06 18:56:37,771 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: hadoop.ssl.server.conf; Ignoring. 2015-05-06 18:56:37,779 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring. 2015-05-06 18:56:37,784 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context mapreduce 2015-05-06 18:56:37,784 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context static 2015-05-06 18:56:37,788 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /mapreduce/* 2015-05-06 18:56:37,788 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /ws/* 2015-05-06 18:56:37,800 INFO [main] org.apache.hadoop.http.HttpServer2: Jetty bound to port 43170 2015-05-06 18:56:37,800 INFO [main] org.mortbay.log: jetty-6.1.26.cloudera.4 2015-05-06 18:56:37,827 INFO [main] org.mortbay.log: Extract jar: file:/opt/cloudera/parcels/CDH-5.3.2-1.cdh5.3.2.p0.10/jars/hadoop-yarn-common-2.5.0-cdh5.3.2.jar!/webapps/mapreduce to /tmp/Jetty_0_0_0_0_43170_mapreduce____.3fyrma/webapp 2015-05-06 18:56:38,170 INFO [main] org.mortbay.log: Started HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:43170 2015-05-06 18:56:38,170 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Web app /mapreduce started at 43170 2015-05-06 18:56:38,496 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Registered webapp guice modules 2015-05-06 18:56:38,499 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator: JOB_CREATE job_1430916274046_0418 2015-05-06 18:56:38,501 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue class java.util.concurrent.LinkedBlockingQueue 2015-05-06 18:56:38,501 INFO Socket Reader #1 for port 34861 org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 34861 2015-05-06 18:56:38,506 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting 2015-05-06 18:56:38,506 INFO [IPC Server listener on 34861] org.apache.hadoop.ipc.Server: IPC Server listener on 34861: starting 2015-05-06 18:56:38,525 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: nodeBlacklistingEnabled:true 2015-05-06 18:56:38,526 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: maxTaskFailuresPerNode is 3 2015-05-06 18:56:38,526 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: blacklistDisablePercent is 33 2015-05-06 18:56:38,596 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: hadoop.ssl.require.client.cert; Ignoring. 2015-05-06 18:56:38,596 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring. 2015-05-06 18:56:38,597 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: hadoop.ssl.client.conf; Ignoring. 2015-05-06 18:56:38,597 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: hadoop.ssl.keystores.factory.class; Ignoring. 2015-05-06 18:56:38,598 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: hadoop.ssl.server.conf; Ignoring. 2015-05-06 18:56:38,602 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring. 2015-05-06 18:56:38,663 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: maxContainerCapability: <memory:202752, vCores:34> 2015-05-06 18:56:38,664 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: queue: root.masterdata 2015-05-06 18:56:38,667 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Upper limit on the thread pool size is 500 2015-05-06 18:56:38,669 INFO [main] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 2015-05-06 18:56:38,698 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1430916274046_0418Job Transitioned from NEW to FAIL_ABORT 2015-05-06 18:56:38,700 INFO CommitterEvent Processor #0 org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_ABORT 2015-05-06 18:56:38,709 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1430916274046_0418Job Transitioned from FAIL_ABORT to FAILED 2015-05-06 18:56:38,709 INFO [Thread-55] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: We are finishing cleanly so this is the last retry 2015-05-06 18:56:38,709 INFO [Thread-55] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify RMCommunicator isAMLastRetry: true 2015-05-06 18:56:38,709 INFO [Thread-55] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: RMCommunicator notified that shouldUnregistered is: true 2015-05-06 18:56:38,709 INFO [Thread-55] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify JHEH isAMLastRetry: true 2015-05-06 18:56:38,709 INFO [Thread-55] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: JobHistoryEventHandler notified that forceJobCompletion is true 2015-05-06 18:56:38,709 INFO [Thread-55] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Calling stop for all the services 2015-05-06 18:56:38,710 INFO [Thread-55] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopping JobHistoryEventHandler. Size of the outstanding queue size is 3 2015-05-06 18:56:38,807 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event Writer setup for JobId: job_1430916274046_0418, File: hdfs://nameservice1/user/devops/.staging/job_1430916274046_0418/job_1430916274046_0418_1.jhist 2015-05-06 18:56:39,144 INFO [Thread-55] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: In stop, writing event JOB_SUBMITTED 2015-05-06 18:56:39,157 INFO [Thread-55] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: In stop, writing event JOB_QUEUE_CHANGED 2015-05-06 18:56:39,157 INFO [Thread-55] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: In stop, writing event JOB_FAILED 2015-05-06 18:56:39,215 INFO [Thread-55] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying hdfs://nameservice1/user/devops/.staging/job_1430916274046_0418/job_1430916274046_0418_1.jhist to hdfs://nameservice1/user/history/done_intermediate/devops/job_1430916274046_0418-1430927787783-devops-Camus+job+%2D+cisgw%2Dmonitoring-1430927798706-0-0-FAILED-root.masterdata-1430927798706.jhist_tmp 2015-05-06 18:56:39,299 INFO [Thread-55] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied to done location: hdfs://nameservice1/user/history/done_intermediate/devops/job_1430916274046_0418-1430927787783-devops-Camus+job+%2D+cisgw%2Dmonitoring-1430927798706-0-0-FAILED-root.masterdata-1430927798706.jhist_tmp 2015-05-06 18:56:39,302 INFO [Thread-55] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying hdfs://nameservice1/user/devops/.staging/job_1430916274046_0418/job_1430916274046_0418_1_conf.xml to hdfs://nameservice1/user/history/done_intermediate/devops/job_1430916274046_0418_conf.xml_tmp 2015-05-06 18:56:39,319 INFO [Thread-55] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied to done location: hdfs://nameservice1/user/history/done_intermediate/devops/job_1430916274046_0418_conf.xml_tmp 2015-05-06 18:56:39,325 INFO [Thread-55] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://nameservice1/user/history/done_intermediate/devops/job_1430916274046_0418.summary_tmp to hdfs://nameservice1/user/history/done_intermediate/devops/job_1430916274046_0418.summary 2015-05-06 18:56:39,327 INFO [Thread-55] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://nameservice1/user/history/done_intermediate/devops/job_1430916274046_0418_conf.xml_tmp to hdfs://nameservice1/user/history/done_intermediate/devops/job_1430916274046_0418_conf.xml 2015-05-06 18:56:39,328 INFO [Thread-55] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://nameservice1/user/history/done_intermediate/devops/job_1430916274046_0418-1430927787783-devops-Camus+job+%2D+cisgw%2Dmonitoring-1430927798706-0-0-FAILED-root.masterdata-1430927798706.jhist_tmp to hdfs://nameservice1/user/history/done_intermediate/devops/job_1430916274046_0418-1430927787783-devops-Camus+job+%2D+cisgw%2Dmonitoring-1430927798706-0-0-FAILED-root.masterdata-1430927798706.jhist 2015-05-06 18:56:39,328 INFO [Thread-55] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopped JobHistoryEventHandler. super.stop() 2015-05-06 18:56:39,329 INFO [Thread-55] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Setting job diagnostics to Job init failed : org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.FileNotFoundException: File does not exist: hdfs://nameservice1/user/devops/.staging/job_1430916274046_0418/job.splitmetainfo at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.createSplits(JobImpl.java:1566) at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transition(JobImpl.java:1430) at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transition(JobImpl.java:1388) at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385) at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302) at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46) at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448) at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:996) at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:138) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:1272) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1045) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1478) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1474) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1407) Caused by: java.io.FileNotFoundException: File does not exist: hdfs://nameservice1/user/devops/.staging/job_1430916274046_0418/job.splitmetainfo at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1093) at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1085) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1085) at org.apache.hadoop.mapreduce.split.SplitMetaInfoReader.readSplitMetaInfo(SplitMetaInfoReader.java:51) at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.createSplits(JobImpl.java:1561) ... 17 more 2015-05-06 18:56:39,331 INFO [Thread-55] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: History url is http://prod-node0138.kyc.myhost.ru:19888/jobhistory/job/job_1430916274046_0418 2015-05-06 18:56:39,339 INFO [Thread-55] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for application to be successfully unregistered. 2015-05-06 18:56:40,340 INFO [Thread-55] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Final Stats: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0 HostLocal:0 RackLocal:0 2015-05-06 18:56:40,341 INFO [Thread-55] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Deleting staging directory hdfs://nameservice1 /user/yarn/.staging/job_1430916274046_0418 2015-05-06 18:56:40,343 INFO [Thread-55] org.apache.hadoop.ipc.Server: Stopping server on 34861 2015-05-06 18:56:40,343 INFO [IPC Server listener on 34861] org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 34861 2015-05-06 18:56:40,346 INFO [TaskHeartbeatHandler PingChecker] org.apache.hadoop.mapreduce.v2.app.TaskHeartbeatHandler: TaskHeartbeatHandler thread interrupted 2015-05-06 18:56:40,346 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: Stopping IPC Server Responder ```
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #850 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/850/)
        MAPREDUCE-3056. Changed the default staging directory to not include user.name to prevent issues with non-secure mode. Contributed by Devaraj K.

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178740
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #850 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/850/ ) MAPREDUCE-3056 . Changed the default staging directory to not include user.name to prevent issues with non-secure mode. Contributed by Devaraj K. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178740 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #29 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/29/)
        MAPREDUCE-3056. svn merge -c r1178740 --ignore-ancestry ../../trunk/

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178741
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #29 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/29/ ) MAPREDUCE-3056 . svn merge -c r1178740 --ignore-ancestry ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178741 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #820 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/820/)
        MAPREDUCE-3056. Changed the default staging directory to not include user.name to prevent issues with non-secure mode. Contributed by Devaraj K.

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178740
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #820 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/820/ ) MAPREDUCE-3056 . Changed the default staging directory to not include user.name to prevent issues with non-secure mode. Contributed by Devaraj K. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178740 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Build #36 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/36/)
        MAPREDUCE-3056. svn merge -c r1178740 --ignore-ancestry ../../trunk/

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178741
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #36 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/36/ ) MAPREDUCE-3056 . svn merge -c r1178740 --ignore-ancestry ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178741 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #1087 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1087/)
        MAPREDUCE-3056. Changed the default staging directory to not include user.name to prevent issues with non-secure mode. Contributed by Devaraj K.

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178740
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1087 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1087/ ) MAPREDUCE-3056 . Changed the default staging directory to not include user.name to prevent issues with non-secure mode. Contributed by Devaraj K. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178740 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #1009 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1009/)
        MAPREDUCE-3056. Changed the default staging directory to not include user.name to prevent issues with non-secure mode. Contributed by Devaraj K.

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178740
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1009 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1009/ ) MAPREDUCE-3056 . Changed the default staging directory to not include user.name to prevent issues with non-secure mode. Contributed by Devaraj K. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178740 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #1029 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1029/)
        MAPREDUCE-3056. Changed the default staging directory to not include user.name to prevent issues with non-secure mode. Contributed by Devaraj K.

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178740
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1029 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1029/ ) MAPREDUCE-3056 . Changed the default staging directory to not include user.name to prevent issues with non-secure mode. Contributed by Devaraj K. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178740 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        Hide
        Vinod Kumar Vavilapalli added a comment -

        +1.

        I just committed this to trunk and branch-0.23. Thanks Devaraj (K)!

        Show
        Vinod Kumar Vavilapalli added a comment - +1. I just committed this to trunk and branch-0.23. Thanks Devaraj (K)!
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12497591/MAPREDUCE-3056-2.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/933//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/933//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12497591/MAPREDUCE-3056-2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/933//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/933//console This message is automatically generated.
        Hide
        Devaraj K added a comment -

        Thanks Vinod.

        You missed changes for mapred-default.xml in the latest patch.

        I,m sorry, missed this. I have included in the latest patch.

        Wrap lines >80 chars

        I have taken care in the updated patch.

        MRAppMaster is a process started by NM and so current-user has no tokens loaded. You can drop the loading of tokens from current-user into the created user.

        Removed the loading of tokens.

        You could collapse MRAppMaster.initAndStartAppMaster() inline and directly invoke main() in the test mocking the System.getenv() calls. Please try if it is possible, otherwise I am okay if you drop this.

        I have not done this change because it requires for the testcase.

        Show
        Devaraj K added a comment - Thanks Vinod. You missed changes for mapred-default.xml in the latest patch. I,m sorry, missed this. I have included in the latest patch. Wrap lines >80 chars I have taken care in the updated patch. MRAppMaster is a process started by NM and so current-user has no tokens loaded. You can drop the loading of tokens from current-user into the created user. Removed the loading of tokens. You could collapse MRAppMaster.initAndStartAppMaster() inline and directly invoke main() in the test mocking the System.getenv() calls. Please try if it is possible, otherwise I am okay if you drop this. I have not done this change because it requires for the testcase.
        Hide
        Vinod Kumar Vavilapalli added a comment -

        You missed changes for mapred-default.xml in the latest patch.

        MRAppMaster:

        • Wrap lines >80 chars
        • MRAppMaster is a process started by NM and so current-user has no tokens loaded. You can drop the loading of tokens from current-user into the created user.
        • You could collapse MRAppMaster.initAndStartAppMaster() inline and directly invoke main() in the test mocking the System.getenv() calls. Please try if it is possible, otherwise I am okay if you drop this.

        Thanks!

        Show
        Vinod Kumar Vavilapalli added a comment - You missed changes for mapred-default.xml in the latest patch. MRAppMaster: Wrap lines >80 chars MRAppMaster is a process started by NM and so current-user has no tokens loaded. You can drop the loading of tokens from current-user into the created user. You could collapse MRAppMaster.initAndStartAppMaster() inline and directly invoke main() in the test mocking the System.getenv() calls. Please try if it is possible, otherwise I am okay if you drop this. Thanks!
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12497470/MAPREDUCE-3056-1.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/926//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/926//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12497470/MAPREDUCE-3056-1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/926//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/926//console This message is automatically generated.
        Hide
        Devaraj K added a comment -

        Thanks for looking into the patch.

        I have updated the patch as per above comments.

        Show
        Devaraj K added a comment - Thanks for looking into the patch. I have updated the patch as per above comments.
        Hide
        Arun C Murthy added a comment -

        Also, don't get user name for config. Always use UGI.getCurrentUser.

        Show
        Arun C Murthy added a comment - Also, don't get user name for config. Always use UGI.getCurrentUser.
        Hide
        Arun C Murthy added a comment -

        Agree with atm. Please provide a test case. Thanks!

        Show
        Arun C Murthy added a comment - Agree with atm. Please provide a test case. Thanks!
        Hide
        Aaron T. Myers added a comment -

        Hey Devaraj, it should be possible to write a test for this, right? Or am I missing something that would make doing so difficult?

        Show
        Aaron T. Myers added a comment - Hey Devaraj, it should be possible to write a test for this, right? Or am I missing something that would make doing so difficult?
        Hide
        Devaraj K added a comment -

        I have verified it manually, jobs submitted by other users are executing fine. Please review this.

        Show
        Devaraj K added a comment - I have verified it manually, jobs submitted by other users are executing fine. Please review this.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12497139/MAPREDUCE-3056.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/906//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/906//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12497139/MAPREDUCE-3056.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/906//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/906//console This message is automatically generated.
        Hide
        Devaraj K added a comment -

        1. The default value of staging dir uses the $

        {user.name}

        . It is getting substituted in the app master with the server user name but it should use the client user name.

        For this issue, as per ATM suggestion I have removed the user.name from the default staging dir.

        2. Here the staging area dir uses the user name. It was using the system user name for this in couple of places and I have changed it to job submitted user name.

        MRApps.java
         
          public static Path getStagingAreaDir(Configuration conf, String user) { 
            return new Path( 
                conf.get(MRJobConfig.MR_AM_STAGING_DIR) + 
                Path.SEPARATOR + user + Path.SEPARATOR + STAGING_CONSTANT); 
          } 
        
        
        Show
        Devaraj K added a comment - 1. The default value of staging dir uses the $ {user.name} . It is getting substituted in the app master with the server user name but it should use the client user name. For this issue, as per ATM suggestion I have removed the user.name from the default staging dir. 2. Here the staging area dir uses the user name. It was using the system user name for this in couple of places and I have changed it to job submitted user name. MRApps.java public static Path getStagingAreaDir(Configuration conf, String user) { return new Path( conf.get(MRJobConfig.MR_AM_STAGING_DIR) + Path.SEPARATOR + user + Path.SEPARATOR + STAGING_CONSTANT); }
        Hide
        Aaron T. Myers added a comment -

        FWIW, basically the same bug exists in 0.20.x when $

        {user.name} is used in the value of the mapred.local.dir with the LinuxTaskController. I'd personally like to completely abandon support for expanding ${user.name}

        in configs for exactly this reason (HADOOP-7522) but in the absence of that, removing it from default values is certainly a step in the right direction.

        Show
        Aaron T. Myers added a comment - FWIW, basically the same bug exists in 0.20.x when $ {user.name} is used in the value of the mapred.local.dir with the LinuxTaskController . I'd personally like to completely abandon support for expanding ${user.name} in configs for exactly this reason ( HADOOP-7522 ) but in the absence of that, removing it from default values is certainly a step in the right direction.
        Hide
        Vinod Kumar Vavilapalli added a comment -

        I ran into this today myself. This happens with DefaultContainerExecutor case, which most of us are not using (surprise )

        Devaraj (K), can you confirm?

        The problem is the default staging dir use of $

        {user.name}

        system property, which is the job-submitter on the client side and the system user on the server side.

        Show
        Vinod Kumar Vavilapalli added a comment - I ran into this today myself. This happens with DefaultContainerExecutor case, which most of us are not using (surprise ) Devaraj (K), can you confirm? The problem is the default staging dir use of $ {user.name} system property, which is the job-submitter on the client side and the system user on the server side.
        Hide
        Devaraj K added a comment -

        Yes Mahadev. App Master is still referring the root's staging dir.

        Client is showing the staging dir as 'hdfs://10.18.52.124:9000/tmp/hadoop-yarn/devaraj/staging/devaraj/.staging' when the submitted user name is 'devaraj'.

        Show
        Devaraj K added a comment - Yes Mahadev. App Master is still referring the root's staging dir. Client is showing the staging dir as 'hdfs://10.18.52.124:9000/tmp/hadoop-yarn/devaraj/staging/devaraj/.staging' when the submitted user name is 'devaraj'.
        Hide
        Mahadev konar added a comment -

        @Devaraj,
        Looks like the staging dir still is pointing to roots dir?

        java.io.FileNotFoundException: File does not exist: hdfs://10.18.52.124:9000/tmp/hadoop-yarn/root/staging/root/.staging/job_1316481926401_0003/job.splitmetainfo

        Do you have the client trace? What staging dir is shown on the client?

        Show
        Mahadev konar added a comment - @Devaraj, Looks like the staging dir still is pointing to roots dir? java.io.FileNotFoundException: File does not exist: hdfs://10.18.52.124:9000/tmp/hadoop-yarn/root/staging/root/.staging/job_1316481926401_0003/job.splitmetainfo Do you have the client trace? What staging dir is shown on the client?
        Hide
        Devaraj K added a comment -
        2011-09-20 10:28:55,661 INFO  impl.JobImpl (JobImpl.java:setup(1001)) - DEBUG --- startJobs: parent=/tmp/hadoop-yarn/root/staging/root/.staging child=job_1316481926401_0003
        2011-09-20 10:28:55,667 INFO  impl.JobImpl (JobImpl.java:setup(1022)) - Adding job token for job_1316481926401_0003 to jobTokenSecretManager
        2011-09-20 10:28:55,987 INFO  impl.JobImpl (JobImpl.java:setup(1038)) - Writing back the job-token file on the remote file system:hdfs://10.18.52.124:9000/tmp/hadoop-yarn/root/staging/root/.staging/job_1316481926401_0003/appTokens
        2011-09-20 10:28:56,486 FATAL app.MRAppMaster (MRAppMaster.java:main(668)) - Error starting MRAppMaster
        org.apache.hadoop.yarn.YarnException: java.io.FileNotFoundException: File does not exist: hdfs://10.18.52.124:9000/tmp/hadoop-yarn/root/staging/root/.staging/job_1316481926401_0003/job.splitmetainfo
        	at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.createSplits(JobImpl.java:1113)
        	at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transition(JobImpl.java:838)
        	at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transition(JobImpl.java:809)
        	at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:376)
        	at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:294)
        	at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:39)
        	at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:439)
        	at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:684)
        	at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:117)
        	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:609)
        	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.start(MRAppMaster.java:563)
        	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:666)
        Caused by: java.io.FileNotFoundException: File does not exist: hdfs://10.18.52.124:9000/tmp/hadoop-yarn/root/staging/root/.staging/job_1316481926401_0003/job.splitmetainfo
        	at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:764)
        	at org.apache.hadoop.mapreduce.split.SplitMetaInfoReader.readSplitMetaInfo(SplitMetaInfoReader.java:50)
        	at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.createSplits(JobImpl.java:1108)
        	... 11 more
        2011-09-20 10:28:56,488 INFO  jobhistory.JobHistoryEventHandler (JobHistoryEventHandler.java:stop(238)) - Stopping JobHistoryEventHandler
        2011-09-20 10:28:56,488 INFO  service.CompositeService (CompositeService.java:stop(97)) - Error stopping JobHistoryEventHandler
        java.lang.NullPointerException
        	at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.stop(JobHistoryEventHandler.java:242)
        	at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:95)
        	at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:85)
        	at org.apache.hadoop.yarn.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:118)
        2011-09-20 10:28:56,489 INFO  service.CompositeService (CompositeService.java:stop(97)) - Error stopping org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter
        java.lang.NullPointerException
        	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter.stop(MRAppMaster.java:487)
        	at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:95)
        	at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:85)
        	at org.apache.hadoop.yarn.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:118)
        2011-09-20 10:28:56,489 INFO  service.CompositeService (CompositeService.java:stop(97)) - Error stopping org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter
        java.lang.NullPointerException
        	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.stop(MRAppMaster.java:443)
        	at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:95)
        	at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:85)
        	at org.apache.hadoop.yarn.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:118)
        2011-09-20 10:28:56,489 INFO  service.CompositeService (CompositeService.java:stop(97)) - Error stopping org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator
        java.lang.IllegalStateException: For this operation, current State must be STARTED instead of INITED
        	at org.apache.hadoop.yarn.service.AbstractService.ensureCurrentState(AbstractService.java:101)
        	at org.apache.hadoop.yarn.service.AbstractService.stop(AbstractService.java:69)
        	at org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator.stop(DefaultSpeculator.java:213)
        	at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:95)
        	at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:85)
        	at org.apache.hadoop.yarn.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:118)
        2011-09-20 10:28:56,491 INFO  service.CompositeService (CompositeService.java:stop(97)) - Error stopping MRClientService
        java.lang.NullPointerException
        	at org.apache.hadoop.mapreduce.v2.app.client.MRClientService.stop(MRClientService.java:161)
        	at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:95)
        	at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:85)
        	at org.apache.hadoop.yarn.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:118)
        2011-09-20 10:28:56,492 INFO  service.CompositeService (CompositeService.java:stop(97)) - Error stopping TaskCleaner
        java.lang.NullPointerException
        	at org.apache.hadoop.mapreduce.v2.app.taskclean.TaskCleanerImpl.stop(TaskCleanerImpl.java:73)
        	at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:95)
        	at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:85)
        	at org.apache.hadoop.yarn.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:118)
        2011-09-20 10:28:56,492 INFO  service.CompositeService (CompositeService.java:stop(97)) - Error stopping org.apache.hadoop.mapred.TaskAttemptListenerImpl
        java.lang.NullPointerException
        	at org.apache.hadoop.mapred.TaskAttemptListenerImpl.stopRpcServer(TaskAttemptListenerImpl.java:128)
        	at org.apache.hadoop.mapred.TaskAttemptListenerImpl.stop(TaskAttemptListenerImpl.java:123)
        	at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:95)
        	at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:85)
        	at org.apache.hadoop.yarn.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:118)
        2011-09-20 10:28:56,492 INFO  service.CompositeService (CompositeService.java:stop(97)) - Error stopping Dispatcher
        java.lang.NullPointerException
        	at org.apache.hadoop.yarn.event.AsyncDispatcher.stop(AsyncDispatcher.java:93)
        	at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:95)
        	at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:85)
        	at org.apache.hadoop.yarn.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:118)
        2011-09-20 10:28:56,493 INFO  service.CompositeService (CompositeService.java:run(120)) - Error stopping org.apache.hadoop.mapreduce.v2.app.MRAppMaster
        java.lang.IllegalStateException: For this operation, current State must be STARTED instead of INITED
        	at org.apache.hadoop.yarn.service.AbstractService.ensureCurrentState(AbstractService.java:101)
        	at org.apache.hadoop.yarn.service.AbstractService.stop(AbstractService.java:69)
        	at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:87)
        	at org.apache.hadoop.yarn.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:118)
        
        Show
        Devaraj K added a comment - 2011-09-20 10:28:55,661 INFO impl.JobImpl (JobImpl.java:setup(1001)) - DEBUG --- startJobs: parent=/tmp/hadoop-yarn/root/staging/root/.staging child=job_1316481926401_0003 2011-09-20 10:28:55,667 INFO impl.JobImpl (JobImpl.java:setup(1022)) - Adding job token for job_1316481926401_0003 to jobTokenSecretManager 2011-09-20 10:28:55,987 INFO impl.JobImpl (JobImpl.java:setup(1038)) - Writing back the job-token file on the remote file system:hdfs://10.18.52.124:9000/tmp/hadoop-yarn/root/staging/root/.staging/job_1316481926401_0003/appTokens 2011-09-20 10:28:56,486 FATAL app.MRAppMaster (MRAppMaster.java:main(668)) - Error starting MRAppMaster org.apache.hadoop.yarn.YarnException: java.io.FileNotFoundException: File does not exist: hdfs://10.18.52.124:9000/tmp/hadoop-yarn/root/staging/root/.staging/job_1316481926401_0003/job.splitmetainfo at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.createSplits(JobImpl.java:1113) at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transition(JobImpl.java:838) at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transition(JobImpl.java:809) at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:376) at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:294) at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:39) at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:439) at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:684) at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:117) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:609) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.start(MRAppMaster.java:563) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:666) Caused by: java.io.FileNotFoundException: File does not exist: hdfs://10.18.52.124:9000/tmp/hadoop-yarn/root/staging/root/.staging/job_1316481926401_0003/job.splitmetainfo at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:764) at org.apache.hadoop.mapreduce.split.SplitMetaInfoReader.readSplitMetaInfo(SplitMetaInfoReader.java:50) at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.createSplits(JobImpl.java:1108) ... 11 more 2011-09-20 10:28:56,488 INFO jobhistory.JobHistoryEventHandler (JobHistoryEventHandler.java:stop(238)) - Stopping JobHistoryEventHandler 2011-09-20 10:28:56,488 INFO service.CompositeService (CompositeService.java:stop(97)) - Error stopping JobHistoryEventHandler java.lang.NullPointerException at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.stop(JobHistoryEventHandler.java:242) at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:95) at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:85) at org.apache.hadoop.yarn.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:118) 2011-09-20 10:28:56,489 INFO service.CompositeService (CompositeService.java:stop(97)) - Error stopping org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter java.lang.NullPointerException at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter.stop(MRAppMaster.java:487) at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:95) at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:85) at org.apache.hadoop.yarn.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:118) 2011-09-20 10:28:56,489 INFO service.CompositeService (CompositeService.java:stop(97)) - Error stopping org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter java.lang.NullPointerException at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.stop(MRAppMaster.java:443) at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:95) at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:85) at org.apache.hadoop.yarn.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:118) 2011-09-20 10:28:56,489 INFO service.CompositeService (CompositeService.java:stop(97)) - Error stopping org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator java.lang.IllegalStateException: For this operation, current State must be STARTED instead of INITED at org.apache.hadoop.yarn.service.AbstractService.ensureCurrentState(AbstractService.java:101) at org.apache.hadoop.yarn.service.AbstractService.stop(AbstractService.java:69) at org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator.stop(DefaultSpeculator.java:213) at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:95) at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:85) at org.apache.hadoop.yarn.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:118) 2011-09-20 10:28:56,491 INFO service.CompositeService (CompositeService.java:stop(97)) - Error stopping MRClientService java.lang.NullPointerException at org.apache.hadoop.mapreduce.v2.app.client.MRClientService.stop(MRClientService.java:161) at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:95) at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:85) at org.apache.hadoop.yarn.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:118) 2011-09-20 10:28:56,492 INFO service.CompositeService (CompositeService.java:stop(97)) - Error stopping TaskCleaner java.lang.NullPointerException at org.apache.hadoop.mapreduce.v2.app.taskclean.TaskCleanerImpl.stop(TaskCleanerImpl.java:73) at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:95) at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:85) at org.apache.hadoop.yarn.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:118) 2011-09-20 10:28:56,492 INFO service.CompositeService (CompositeService.java:stop(97)) - Error stopping org.apache.hadoop.mapred.TaskAttemptListenerImpl java.lang.NullPointerException at org.apache.hadoop.mapred.TaskAttemptListenerImpl.stopRpcServer(TaskAttemptListenerImpl.java:128) at org.apache.hadoop.mapred.TaskAttemptListenerImpl.stop(TaskAttemptListenerImpl.java:123) at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:95) at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:85) at org.apache.hadoop.yarn.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:118) 2011-09-20 10:28:56,492 INFO service.CompositeService (CompositeService.java:stop(97)) - Error stopping Dispatcher java.lang.NullPointerException at org.apache.hadoop.yarn.event.AsyncDispatcher.stop(AsyncDispatcher.java:93) at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:95) at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:85) at org.apache.hadoop.yarn.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:118) 2011-09-20 10:28:56,493 INFO service.CompositeService (CompositeService.java:run(120)) - Error stopping org.apache.hadoop.mapreduce.v2.app.MRAppMaster java.lang.IllegalStateException: For this operation, current State must be STARTED instead of INITED at org.apache.hadoop.yarn.service.AbstractService.ensureCurrentState(AbstractService.java:101) at org.apache.hadoop.yarn.service.AbstractService.stop(AbstractService.java:69) at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:87) at org.apache.hadoop.yarn.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:118)

          People

          • Assignee:
            Devaraj K
            Reporter:
            Devaraj K
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development