[FLINK-9011] YarnResourceManager spamming log file at INFO level

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.5.0
• Fix Version/s: 1.5.0
• Component/s: Deployment / YARN, Runtime / Coordination
• Labels: flip-6
• Assignee: vinoyang
• Reporter: Nico Kruber
• Created: 16/Mar/18 14:58
• Updated: 28/Feb/19 14:00
• Resolved: 18/Apr/18 11:17

Description

For every requested resource, the YarnResourceManager spams the log with log-level INFO and the following messages:

2018-03-16 03:41:20,180 INFO  org.apache.flink.yarn.YarnResourceManager                     - Received new container: container_1521038088305_0257_01_000102 - Remaining pending container requests: 301
2018-03-16 03:41:20,180 INFO  org.apache.flink.yarn.YarnResourceManager                     - TaskExecutor container_1521038088305_0257_01_000102 will be started with container size 8192 MB, JVM heap size 5120 MB, JVM direct memory limit 3072 MB
2018-03-16 03:41:20,180 INFO  org.apache.flink.yarn.YarnResourceManager                     - TM:remote keytab path obtained null
2018-03-16 03:41:20,180 INFO  org.apache.flink.yarn.YarnResourceManager                     - TM:remote keytab principal obtained null
2018-03-16 03:41:20,180 INFO  org.apache.flink.yarn.YarnResourceManager                     - TM:remote yarn conf path obtained null
2018-03-16 03:41:20,180 INFO  org.apache.flink.yarn.YarnResourceManager                     - TM:remote krb5 path obtained null
2018-03-16 03:41:20,181 INFO  org.apache.flink.yarn.Utils                                   - Copying from file:/mnt/yarn/usercache/hadoop/appcache/application_1521038088305_0257/container_1521038088305_0257_01_000001/6766be70-82f7-4999-a371-11c27527fb6e-taskmanager-conf.yaml to hdfs://ip-172-31-1-91.eu-west-1.compute.internal:8020/user/hadoop/.flink/application_1521038088305_0257/6766be70-82f7-4999-a371-11c27527fb6e-taskmanager-conf.yaml
2018-03-16 03:41:20,190 INFO  org.apache.flink.yarn.YarnResourceManager                     - Prepared local resource for modified yaml: resource { scheme: "hdfs" host: "ip-172-31-1-91.eu-west-1.compute.internal" port: 8020 file: "/user/hadoop/.flink/application_1521038088305_0257/6766be70-82f7-4999-a371-11c27527fb6e-taskmanager-conf.yaml" } size: 595 timestamp: 1521171680190 type: FILE visibility: APPLICATION
2018-03-16 03:41:20,194 INFO  org.apache.flink.yarn.YarnResourceManager                     - Creating container launch context for TaskManagers
2018-03-16 03:41:20,194 INFO  org.apache.flink.yarn.YarnResourceManager                     - Starting TaskManagers with command: $JAVA_HOME/bin/java -Xms5120m -Xmx5120m -XX:MaxDirectMemorySize=3072m  -Dlog.file=<LOG_DIR>/taskmanager.log -Dlogback.configurationFile=file:./logback.xml -Dlog4j.configuration=file:./log4j.properties org.apache.flink.yarn.YarnTaskExecutorRunner --configDir . 1> <LOG_DIR>/taskmanager.out 2> <LOG_DIR>/taskmanager.err

Issue Links

• links to: GitHub Pull Request #5712