[FLINK-20143] use `yarn.provided.lib.dirs` config deploy job failed in yarn per job mode

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.11.2, 1.12.0
• Fix Version/s: 1.11.3, 1.12.0
• Component/s: Client / Job Submission, Deployment / YARN
• Labels: pull-request-available
• Assignee: Yang Wang
• Reporter: zhisheng
• Created: 13/Nov/20 08:24
• Updated: 18/Nov/20 10:06
• Resolved: 18/Nov/20 10:06

Description

use follow command deploy flink job to yarn failed 

./bin/flink run -m yarn-cluster -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib    ./examples/streaming/StateMachineExample.jar

log:

$ ./bin/flink run -m yarn-cluster -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib    ./examples/streaming/StateMachineExample.jar$ ./bin/flink run -m yarn-cluster -d -ynm flink-1.12-test -ytm 3g -yjm 3g -yD yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib    ./examples/streaming/StateMachineExample.jarSLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/data1/app/flink-1.12-SNAPSHOT/lib/log4j-slf4j-impl-2.12.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/data1/app/hadoop-2.7.3-snappy-32core12disk/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/data1/app/hadoop-2.7.3-snappy-32core12disk/share/hadoop/tools/lib/hadoop-aliyun-2.9.2-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]2020-11-13 16:14:30,347 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                [] - Dynamic Property set: yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib2020-11-13 16:14:30,347 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                [] - Dynamic Property set: yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/libUsage with built-in data generator: StateMachineExample [--error-rate <probability-of-invalid-transition>] [--sleep <sleep-per-record-in-ms>]Usage with Kafka: StateMachineExample --kafka-topic <topic> [--brokers <brokers>]Options for both the above setups: [--backend <file|rocks>] [--checkpoint-dir <filepath>] [--async-checkpoints <true|false>] [--incremental-checkpoints <true|false>] [--output <filepath> OR null for stdout]
Using standalone source with error rate 0.000000 and sleep delay 1 millis
2020-11-13 16:14:30,706 WARN  org.apache.flink.yarn.configuration.YarnLogConfigUtil        [] - The configuration directory ('/data1/app/flink-1.12-SNAPSHOT/conf') already contains a LOG4J config file.If you want to use logback, then please delete or rename the log configuration file.2020-11-13 16:14:30,947 INFO  org.apache.hadoop.yarn.client.AHSProxy                       [] - Connecting to Application History server at FAT-hadoopuat-69117.vm.dc01.tech/10.69.1.17:102002020-11-13 16:14:30,958 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar2020-11-13 16:14:31,065 INFO  org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider [] - Failing over to rm22020-11-13 16:14:31,130 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - The configured JobManager memory is 3072 MB. YARN will allocate 4096 MB to make up an integer multiple of its minimum allocation memory (2048 MB, configured via 'yarn.scheduler.minimum-allocation-mb'). The extra 1024 MB may not be used by Flink.2020-11-13 16:14:31,130 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - The configured TaskManager memory is 3072 MB. YARN will allocate 4096 MB to make up an integer multiple of its minimum allocation memory (2048 MB, configured via 'yarn.scheduler.minimum-allocation-mb'). The extra 1024 MB may not be used by Flink.2020-11-13 16:14:31,130 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Cluster specification: ClusterSpecification{masterMemoryMB=3072, taskManagerMemoryMB=3072, slotsPerTaskManager=2}2020-11-13 16:14:31,681 WARN  org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory      [] - The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.2020-11-13 16:14:33,417 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Submitting application master application_1599741232083_219902020-11-13 16:14:33,446 INFO  org.apache.hadoop.yarn.client.api.impl.YarnClientImpl        [] - Submitted application application_1599741232083_219902020-11-13 16:14:33,446 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Waiting for the cluster to be allocated2020-11-13 16:14:33,448 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Deploying cluster, current state ACCEPTED
------------------------------------------------------------ The program finished with the following exception:
org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: Could not deploy Yarn job cluster. at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:330) at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:198) at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114) at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:743) at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:242) at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:971) at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1047) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1047)Caused by: org.apache.flink.client.deployment.ClusterDeploymentException: Could not deploy Yarn job cluster. at org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:460) at org.apache.flink.client.deployment.executors.AbstractJobClusterExecutor.execute(AbstractJobClusterExecutor.java:70) at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:1916) at org.apache.flink.client.program.StreamContextEnvironment.executeAsync(StreamContextEnvironment.java:128) at org.apache.flink.client.program.StreamContextEnvironment.execute(StreamContextEnvironment.java:76) at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1798) at org.apache.flink.streaming.examples.statemachine.StateMachineExample.main(StateMachineExample.java:142) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:316) ... 11 moreCaused by: org.apache.flink.yarn.YarnClusterDescriptor$YarnDeploymentException: The YARN application unexpectedly switched to state FAILED during deployment.Diagnostics from YARN: Application application_1599741232083_21990 failed 2 times in previous 10000 milliseconds due to AM Container for appattempt_1599741232083_21990_000002 exited with  exitCode: -1Failing this attempt.Diagnostics: [2020-11-13 16:14:38.244]Destination must be relativeFor more detailed output, check the application tracking page: http://FAT-hadoopuat-69117.vm.dc01.tech:8188/applicationhistory/app/application_1599741232083_21990 Then click on links to logs of each attempt.. Failing the application.If log aggregation is enabled on your cluster, use this command to further investigate the issue:yarn logs -applicationId application_1599741232083_21990 at org.apache.flink.yarn.YarnClusterDescriptor.startAppMaster(YarnClusterDescriptor.java:1078) at org.apache.flink.yarn.YarnClusterDescriptor.deployInternal(YarnClusterDescriptor.java:558) at org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:453) ... 22 more2020-11-13 16:14:38,492 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Cancelling deployment from Deployment Failure Hook2020-11-13 16:14:38,494 INFO  org.apache.hadoop.yarn.client.AHSProxy                       [] - Connecting to Application History server at FAT-hadoopuat-69117.vm.dc01.tech/10.69.1.17:102002020-11-13 16:14:38,495 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Killing YARN application2020-11-13 16:14:38,499 INFO  org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider [] - Failing over to rm22020-11-13 16:14:38,503 INFO  org.apache.hadoop.yarn.client.api.impl.YarnClientImpl        [] - Killed application application_1599741232083_219902020-11-13 16:14:38,503 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Deleting files in hdfs://flashHadoopUAT/user/deploy/.flink/application_1599741232083_21990.

but if i set `execution.target: yarn-per-job` in flink-conf.yaml, it runs ok

if i run in application mode, it runs ok too

./bin/flink run-application -p 2 -d -t yarn-application -ytm 3g -yjm 3g -yD yarn.provided.lib.dirs=hdfs:///flink/flink-1.12-SNAPSHOT/lib ./examples/streaming/StateMachineExample.jar

but the jobid is 00000000000000000000000000000000

 

 

 

Issue Links

• links to: GitHub Pull Request #14082