[FLINK-5207] Decrease HadoopFileSystem logging

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.1.4, 1.2.0
• Assignee: Unassigned
• Reporter: Ufuk Celebi
• Created: 29/Nov/16 16:02
• Updated: 01/Dec/16 09:51
• Resolved: 01/Dec/16 09:51

Description

HadoopFileSystem logging is very noisy when trying to get the Hadoop configuration, which happens for every HadoopFileSystem. This happens for example when a checkpoint is written to HDFS in case of HA.

1) Two log messages relate to the user config:

final String hdfsDefaultPath = GlobalConfiguration.getString(ConfigConstants.HDFS_DEFAULT_CONFIG, null);
		if (hdfsDefaultPath != null) {
			retConf.addResource(new org.apache.hadoop.fs.Path(hdfsDefaultPath));
		} else {
			LOG.debug("Cannot find hdfs-default configuration file");
		}

Since this is reflected in the logged configuration properties, it is mostly unnecessary to log over and over again. Therefore I propose to go with TRACE.

(2) Two log messages relate to the default config files of Hadoop. There I would change the debug message to happen only when they are not found. If it is still too noisy, we could completely remove it or decrease to TRACE at another time.