Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-5207

Decrease HadoopFileSystem logging

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.2.0, 1.1.4
    • Component/s: None
    • Labels:
      None

      Description

      HadoopFileSystem logging is very noisy when trying to get the Hadoop configuration, which happens for every HadoopFileSystem. This happens for example when a checkpoint is written to HDFS in case of HA.

      1) Two log messages relate to the user config:

      final String hdfsDefaultPath = GlobalConfiguration.getString(ConfigConstants.HDFS_DEFAULT_CONFIG, null);
      		if (hdfsDefaultPath != null) {
      			retConf.addResource(new org.apache.hadoop.fs.Path(hdfsDefaultPath));
      		} else {
      			LOG.debug("Cannot find hdfs-default configuration file");
      		}
      

      Since this is reflected in the logged configuration properties, it is mostly unnecessary to log over and over again. Therefore I propose to go with TRACE.

      (2) Two log messages relate to the default config files of Hadoop. There I would change the debug message to happen only when they are not found. If it is still too noisy, we could completely remove it or decrease to TRACE at another time.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              uce Ufuk Celebi
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: