Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6736

import hive table from parquet files, there is no 'job.splitmetainfo' file message

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Blocker
    • Resolution: Unresolved
    • Affects Version/s: 2.6.0
    • Fix Version/s: None
    • Component/s: applicationmaster, mrv2
    • Labels:
      None
    • Environment:

      Ubuntu 14.04.4 LTS (GNU/Linux 4.2.0-27-generic x86_64)
      Hadoop 2.6.0-cdh5.7.0

      Description

      same issue : https://issues.apache.org/jira/browse/MAPREDUCE-3056

      which is created on 2011.09.21 and fixed 2011.10.04
      there is user(Sergey) who has same issue on 2015.05.13 too! (look last comment of above link)

      on beeline prompt,
      I've tried import hive table from parquet files that are exported from another hive table
      but there was no "job.splitmetainfo" file on staging directory. So FileNotFoundException occurs

      I tried with yarn user, hdfs user with different permissions and directory ownership, but nothing changed.

      full log messages below

      ==================

      Log Type: syslog
      Log Upload Time: Wed Jul 20 17:57:36 +0900 2016
      Log Length: 21439
      2016-07-20 17:57:26,139 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1468834620182_0036_000001
      2016-07-20 17:57:26,417 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
      2016-07-20 17:57:26,463 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
      2016-07-20 17:57:26,463 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN, Service: , Ident: (org.apache.hadoop.yarn.security.AMRMTokenIdentifier@52af26ee)
      2016-07-20 17:57:26,510 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Using mapred newApiCommitter.
      2016-07-20 17:57:26,991 WARN [main] org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
      2016-07-20 17:57:27,091 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config null
      2016-07-20 17:57:27,154 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1
      2016-07-20 17:57:27,159 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter is org.apache.hadoop.tools.mapred.CopyCommitter
      2016-07-20 17:57:27,231 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.jobhistory.EventType for class org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler
      2016-07-20 17:57:27,232 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher
      2016-07-20 17:57:27,233 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher
      2016-07-20 17:57:27,234 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher
      2016-07-20 17:57:27,235 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler
      2016-07-20 17:57:27,241 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.speculate.Speculator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$SpeculatorEventDispatcher
      2016-07-20 17:57:27,242 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter
      2016-07-20 17:57:27,243 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncher$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter
      2016-07-20 17:57:27,292 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://da74:8020]
      2016-07-20 17:57:27,322 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://da74:8020]
      2016-07-20 17:57:27,352 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://da74:8020]
      2016-07-20 17:57:27,382 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Perms after creating 448, Expected: 448
      2016-07-20 17:57:27,387 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Emitting job history data to the timeline server is not enabled
      2016-07-20 17:57:27,428 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler
      2016-07-20 17:57:27,645 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
      2016-07-20 17:57:27,703 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
      2016-07-20 17:57:27,704 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MRAppMaster metrics system started
      2016-07-20 17:57:27,712 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Adding job token for job_1468834620182_0036 to jobTokenSecretManager
      2016-07-20 17:57:27,723 WARN [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Job init failed
      org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.FileNotFoundException: File does not exist: hdfs://da74:8020/user/yarn/staging/hdfs/.staging/job_1468834620182_0036/job.splitmetainfo
      at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.createSplits(JobImpl.java:1579)
      at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transition(JobImpl.java:1443)
      at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transition(JobImpl.java:1401)
      at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
      at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
      at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
      at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
      at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:997)
      at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:139)
      at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:1333)
      at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1101)
      at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
      at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java:1544)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:422)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
      at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1540)
      at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1473)
      Caused by: java.io.FileNotFoundException: File does not exist: hdfs://da74:8020/user/yarn/staging/hdfs/.staging/job_1468834620182_0036/job.splitmetainfo
      at org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1219)
      at org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1211)
      at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
      at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1211)
      at org.apache.hadoop.mapreduce.split.SplitMetaInfoReader.readSplitMetaInfo(SplitMetaInfoReader.java:51)
      at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.createSplits(JobImpl.java:1574)
      ... 17 more
      2016-07-20 17:57:27,726 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster launching normal, non-uberized, multi-container job job_1468834620182_0036.
      2016-07-20 17:57:27,753 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue class java.util.concurrent.LinkedBlockingQueue
      2016-07-20 17:57:27,761 INFO Socket Reader #1 for port 35855 org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 35855
      2016-07-20 17:57:27,778 INFO [main] org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding protocol org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB to the server
      2016-07-20 17:57:27,779 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting
      2016-07-20 17:57:27,779 INFO [IPC Server listener on 35855] org.apache.hadoop.ipc.Server: IPC Server listener on 35855: starting
      2016-07-20 17:57:27,780 INFO [main] org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Instantiated MRClientService at da74/115.68.67.98:35855
      2016-07-20 17:57:27,837 INFO [main] org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
      2016-07-20 17:57:27,842 INFO [main] org.apache.hadoop.security.authentication.server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets.
      2016-07-20 17:57:27,846 INFO [main] org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.mapreduce is not defined
      2016-07-20 17:57:27,855 INFO [main] org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
      2016-07-20 17:57:27,860 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context mapreduce
      2016-07-20 17:57:27,860 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context static
      2016-07-20 17:57:27,862 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /mapreduce/*
      2016-07-20 17:57:27,862 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /ws/*
      2016-07-20 17:57:27,870 INFO [main] org.apache.hadoop.http.HttpServer2: Jetty bound to port 42206
      2016-07-20 17:57:27,870 INFO [main] org.mortbay.log: jetty-6.1.26.cloudera.4
      2016-07-20 17:57:27,899 INFO [main] org.mortbay.log: Extract jar:file:/usr/lib/hadoop-yarn/hadoop-yarn-common-2.6.0-cdh5.7.0.jar!/webapps/mapreduce to /tmp/Jetty_0_0_0_0_42206_mapreduce____p5x30f/webapp
      2016-07-20 17:57:28,159 INFO [main] org.mortbay.log: Started HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:42206
      2016-07-20 17:57:28,160 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Web app /mapreduce started at 42206
      2016-07-20 17:57:28,493 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Registered webapp guice modules
      2016-07-20 17:57:28,497 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator: JOB_CREATE job_1468834620182_0036
      2016-07-20 17:57:28,499 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue class java.util.concurrent.LinkedBlockingQueue
      2016-07-20 17:57:28,499 INFO Socket Reader #1 for port 41561 org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 41561
      2016-07-20 17:57:28,506 INFO [IPC Server listener on 41561] org.apache.hadoop.ipc.Server: IPC Server listener on 41561: starting
      2016-07-20 17:57:28,502 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting
      2016-07-20 17:57:28,540 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: nodeBlacklistingEnabled:true
      2016-07-20 17:57:28,540 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: maxTaskFailuresPerNode is 3
      2016-07-20 17:57:28,540 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: blacklistDisablePercent is 33
      2016-07-20 17:57:28,605 INFO [main] org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at da74/115.68.67.98:8030
      2016-07-20 17:57:28,687 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: maxContainerCapability: <memory:8192, vCores:4>
      2016-07-20 17:57:28,687 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: queue: root.hdfs
      2016-07-20 17:57:28,692 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Upper limit on the thread pool size is 500
      2016-07-20 17:57:28,692 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: The thread pool initial size is 10
      2016-07-20 17:57:28,694 INFO [main] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0
      2016-07-20 17:57:28,699 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1468834620182_0036Job Transitioned from NEW to FAIL_ABORT
      2016-07-20 17:57:28,700 INFO CommitterEvent Processor #0 org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_ABORT
      2016-07-20 17:57:28,727 INFO CommitterEvent Processor #0 org.apache.hadoop.tools.mapred.CopyCommitter: Cleaning up temporary work folder: /user/hive/staging/hdfs/.staging/_distcp70301392
      2016-07-20 17:57:28,758 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1468834620182_0036Job Transitioned from FAIL_ABORT to FAILED
      2016-07-20 17:57:28,759 INFO [Thread-55] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: We are finishing cleanly so this is the last retry
      2016-07-20 17:57:28,759 INFO [Thread-55] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify RMCommunicator isAMLastRetry: true
      2016-07-20 17:57:28,759 INFO [Thread-55] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: RMCommunicator notified that shouldUnregistered is: true
      2016-07-20 17:57:28,760 INFO [Thread-55] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify JHEH isAMLastRetry: true
      2016-07-20 17:57:28,760 INFO [Thread-55] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: JobHistoryEventHandler notified that forceJobCompletion is true
      2016-07-20 17:57:28,760 INFO [Thread-55] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Calling stop for all the services
      2016-07-20 17:57:28,760 INFO [Thread-55] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopping JobHistoryEventHandler. Size of the outstanding queue size is 3
      2016-07-20 17:57:28,850 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event Writer setup for JobId: job_1468834620182_0036, File: hdfs://da74:8020/user/yarn/staging/hdfs/.staging/job_1468834620182_0036/job_1468834620182_0036_1.jhist
      2016-07-20 17:57:29,188 INFO [Thread-55] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: In stop, writing event JOB_SUBMITTED
      2016-07-20 17:57:29,223 INFO [Thread-55] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://da74:8020]
      2016-07-20 17:57:29,253 INFO [Thread-55] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: In stop, writing event JOB_QUEUE_CHANGED
      2016-07-20 17:57:29,253 INFO [Thread-55] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: In stop, writing event JOB_FAILED
      2016-07-20 17:57:29,376 INFO [Thread-55] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying hdfs://da74:8020/user/yarn/staging/hdfs/.staging/job_1468834620182_0036/job_1468834620182_0036_1.jhist to hdfs://da74:8020/user/yarn/staging/history/done_intermediate/hdfs/job_1468834620182_0036-1469005044142-hdfs-distcp-1469005048754-0-0-FAILED-root.hdfs-1469005048754.jhist_tmp
      2016-07-20 17:57:29,513 INFO [Thread-55] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied to done location: hdfs://da74:8020/user/yarn/staging/history/done_intermediate/hdfs/job_1468834620182_0036-1469005044142-hdfs-distcp-1469005048754-0-0-FAILED-root.hdfs-1469005048754.jhist_tmp
      2016-07-20 17:57:29,520 INFO [Thread-55] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying hdfs://da74:8020/user/yarn/staging/hdfs/.staging/job_1468834620182_0036/job_1468834620182_0036_1_conf.xml to hdfs://da74:8020/user/yarn/staging/history/done_intermediate/hdfs/job_1468834620182_0036_conf.xml_tmp
      2016-07-20 17:57:29,561 INFO [Thread-55] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied to done location: hdfs://da74:8020/user/yarn/staging/history/done_intermediate/hdfs/job_1468834620182_0036_conf.xml_tmp
      2016-07-20 17:57:29,580 INFO [Thread-55] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://da74:8020/user/yarn/staging/history/done_intermediate/hdfs/job_1468834620182_0036.summary_tmp to hdfs://da74:8020/user/yarn/staging/history/done_intermediate/hdfs/job_1468834620182_0036.summary
      2016-07-20 17:57:29,586 INFO [Thread-55] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://da74:8020/user/yarn/staging/history/done_intermediate/hdfs/job_1468834620182_0036_conf.xml_tmp to hdfs://da74:8020/user/yarn/staging/history/done_intermediate/hdfs/job_1468834620182_0036_conf.xml
      2016-07-20 17:57:29,594 INFO [Thread-55] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://da74:8020/user/yarn/staging/history/done_intermediate/hdfs/job_1468834620182_0036-1469005044142-hdfs-distcp-1469005048754-0-0-FAILED-root.hdfs-1469005048754.jhist_tmp to hdfs://da74:8020/user/yarn/staging/history/done_intermediate/hdfs/job_1468834620182_0036-1469005044142-hdfs-distcp-1469005048754-0-0-FAILED-root.hdfs-1469005048754.jhist
      2016-07-20 17:57:29,594 INFO [Thread-55] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopped JobHistoryEventHandler. super.stop()
      2016-07-20 17:57:29,596 INFO [Thread-55] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Setting job diagnostics to Job init failed : org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.FileNotFoundException: File does not exist: hdfs://da74:8020/user/yarn/staging/hdfs/.staging/job_1468834620182_0036/job.splitmetainfo
      at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.createSplits(JobImpl.java:1579)
      at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transition(JobImpl.java:1443)
      at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transition(JobImpl.java:1401)
      at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
      at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
      at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
      at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
      at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:997)
      at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:139)
      at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:1333)
      at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1101)
      at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
      at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java:1544)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:422)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
      at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1540)
      at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1473)
      Caused by: java.io.FileNotFoundException: File does not exist: hdfs://da74:8020/user/yarn/staging/hdfs/.staging/job_1468834620182_0036/job.splitmetainfo
      at org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1219)
      at org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1211)
      at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
      at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1211)
      at org.apache.hadoop.mapreduce.split.SplitMetaInfoReader.readSplitMetaInfo(SplitMetaInfoReader.java:51)
      at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.createSplits(JobImpl.java:1574)
      ... 17 more

      2016-07-20 17:57:29,597 INFO [Thread-55] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: History url is http://da74:19888/jobhistory/job/job_1468834620182_0036
      2016-07-20 17:57:29,604 INFO [Thread-55] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for application to be successfully unregistered.
      2016-07-20 17:57:30,605 INFO [Thread-55] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Final Stats: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0 HostLocal:0 RackLocal:0
      2016-07-20 17:57:30,607 INFO [Thread-55] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Deleting staging directory hdfs://da74:8020/ /user/hive/staging/hdfs/.staging/job_1468834620182_0036
      2016-07-20 17:57:30,628 INFO [Thread-55] org.apache.hadoop.ipc.Server: Stopping server on 41561
      2016-07-20 17:57:30,629 INFO [IPC Server listener on 41561] org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 41561
      2016-07-20 17:57:30,629 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: Stopping IPC Server Responder
      2016-07-20 17:57:30,629 INFO [TaskHeartbeatHandler PingChecker] org.apache.hadoop.mapreduce.v2.app.TaskHeartbeatHandler: TaskHeartbeatHandler thread interrupted
      2016-07-20 17:57:30,629 INFO [Ping Checker] org.apache.hadoop.yarn.util.AbstractLivelinessMonitor: TaskAttemptFinishingMonitor thread interrupted

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              curycu Ha, Hun Cheol
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: