[TEZ-4175] Consider removing YarnConfiguration where it's possible

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.10.0
• Assignee: László Bodor
• Reporter: László Bodor
• Created: 09/May/20 09:15
• Updated: 26/Aug/20 18:05
• Resolved: 26/Aug/20 18:05

Description

A comment in DAGAppmaster made me think that we don't need to rely on YarnConfiguration in all cases, what if it can be replace with base Configuration...

      // TODO Does this really need to be a YarnConfiguration ?
      Configuration conf = new Configuration(new YarnConfiguration());

In hadoop 3.1.3 source, I cannot see that it adds e.g. yarn-site as a resource:

  public YarnConfiguration() {
    super();
  }
  
  public YarnConfiguration(Configuration conf) {
    super(conf);
    if (! (conf instanceof YarnConfiguration)) {
      this.reloadConfiguration();
    }
  }

UPDATE: YarnConfiguration adds yarn-site as a default resource in a static initializer, so it should be used when yarn configs are expected to be present

in current codebase:

grep -iRH "new YarnConfiguration" --include="*.java"

tez-plugins/tez-history-parser/src/main/java/org/apache/tez/history/ATSImportTool.java:    YarnConfiguration yarnConf = new YarnConfiguration(conf);
tez-plugins/tez-aux-services/src/main/java/org/apache/tez/auxservices/ShuffleHandler.java:    super.serviceInit(new YarnConfiguration(conf));
tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java:    YarnConfiguration yarnConf = new YarnConfiguration(tezConf);
tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java:    YarnConfiguration yarnConf = new YarnConfiguration(tezConf);
tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java:    YarnConfiguration yarnConf = new YarnConfiguration(tezConf);
tez-api/src/test/java/org/apache/tez/client/TestTezClient.java:    tezClient.init(new TezConfiguration(false), new YarnConfiguration());
tez-api/src/main/java/org/apache/tez/client/TezClient.java:    amConfig.setYarnConfiguration(new YarnConfiguration(amConfig.getTezConfiguration()));
tez-api/src/main/java/org/apache/tez/client/TezClient.java:    amConfig.setYarnConfiguration(new YarnConfiguration(amConfig.getTezConfiguration()));
tez-api/src/main/java/org/apache/tez/client/TezClient.java:    return getDAGClient(appId, tezConf, new YarnConfiguration(tezConf), frameworkClient, ugi);
tez-tests/src/test/java/org/apache/tez/test/FaultToleranceTestRunner.java:      tezConf = new TezConfiguration(new YarnConfiguration());
tez-tests/src/test/java/org/apache/tez/test/FaultToleranceTestRunner.java:       tezConf = new TezConfiguration(new YarnConfiguration(this.conf));
tez-mapreduce/src/test/java/org/apache/tez/mapreduce/hadoop/TestMRInputHelpers.java:    Configuration testConf = new YarnConfiguration(
tez-mapreduce/src/main/java/org/apache/tez/mapreduce/client/YARNRunner.java:   this(conf, new ResourceMgrDelegate(new YarnConfiguration(conf)));
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:    Configuration conf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:    Configuration conf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:    Configuration tezConf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:    Configuration tezConf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:    Configuration tezConf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:    Configuration tezConf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:    Configuration tezConf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:    Configuration tezConf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:    Configuration tezConf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:    Configuration tezConf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:    Configuration tezConf = new Configuration(new YarnConfiguration());
tez-dag/src/main/java/org/apache/tez/dag/app/DAGAppMaster.java:      Configuration conf = new Configuration(new YarnConfiguration());

Issue Links

• is related to: TEZ-1391 IGNORE_LIB_URIS is not taking affect in local mode