[REEF-348] Job localization error caused by incorrect timestamp on HDInsight

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Component/s: REEF-Runtime-HDInsight, REEF.NET

Environment

REEF.NET on HDInsight

• Assignee: Andrew Chung
• Reporter: Andrew Chung
• Created: 26/May/15 21:19
• Updated: 12/Oct/17 09:33
• Resolved: 12/Oct/17 09:33

Description

The timestamp is set incorrectly when uploading global.jar to blob, so the resource could not be successfully localized. Below are nodemanager logs:

2015-05-21 03:04:16,391 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Copying from c:/apps/temp/hdfs/nm-local-dir/nmPrivate/container_1432083697923_0483_01_000024.tokens to c:/apps/temp/hdfs/nm-local-dir/usercache/gopher/appcache/application_1432083697923_0483/container_1432083697923_0483_01_000024.tokens
2015-05-21 03:04:16,391 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: CWD set to c:/apps/temp/hdfs/nm-local-dir/usercache/gopher/appcache/application_1432083697923_0483 = file:/c:/apps/temp/hdfs/nm-local-dir/usercache/gopher/appcache/application_1432083697923_0483
2015-05-21 03:04:16,737 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource wasb://[e-mail]/NRT/c6cbe03a-0ce1-4952-918a-f4cebac4f9ed/reef-evaluator-3911340348394906440.jar(->file:/c:/apps/temp/hdfs/nm-local-dir/usercache/gopher/appcache/application_1432083697923_0483/filecache/22/reef-evaluator-3911340348394906440.jar) transitioned from DOWNLOADING to LOCALIZED
2015-05-21 03:04:16,752 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: DEBUG: FAILED { wasb://[e-mail]/NRT/c6cbe03a-0ce1-4952-918a-f4cebac4f9ed/global.jar, 1432177412000, ARCHIVE, null }, Resource wasb://[e-mail]/NRT/c6cbe03a-0ce1-4952-918a-f4cebac4f9ed/global.jar changed on src filesystem (expected 1432177412000, was 1432177417000
2015-05-21 03:04:16,767 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource wasb://[e-mail]/NRT/c6cbe03a-0ce1-4952-918a-f4cebac4f9ed/global.jar transitioned from DOWNLOADING to FAILED
2015-05-21 03:04:16,767 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1432083697923_0483_01_000024 transitioned from LOCALIZING to LOCALIZATION_FAILED

Issue Links

• links to: Link to PR