[YARN-9875] FSSchedulerConfigurationStore fails to update with hdfs path

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.0, 3.2.3
• Component/s: capacityscheduler
• Labels: pull-request-available
• Assignee: Prabhu Joseph
• Reporter: Prabhu Joseph
• Created: 06/Oct/19 17:07
• Updated: 04/Aug/21 07:13
• Resolved: 19/Oct/19 00:32

Description

FSSchedulerConfigurationStore fails to update with hdfs path - "java.io.IOException: Filesystem closed"

RM Logs:

2019-10-06 16:50:40,829 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.conf.FSSchedulerConfigurationStore: write temp capacity configuration fail, schedulerConfigFile=hdfs:/tmp/yarn/system/capacity-scheduler.xml.1570380640828.tmp
java.io.IOException: Filesystem closed
        at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:475)
        at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1232)
        at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1214)
        at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1196)
        at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1134)
        at org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:530)
        at org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:527)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:541)
        at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:468)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1136)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1116)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1005)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:993)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.conf.FSSchedulerConfigurationStore.writeTmpConfig(FSSchedulerConfigurationStore.java:251)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.conf.FSSchedulerConfigurationStore.logMutation(FSSchedulerConfigurationStore.java:130)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.conf.MutableCSConfigurationProvider.logAndApplyMutation(MutableCSConfigurationProvider.java:153)
        at org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices$13.run(RMWebServices.java:2597)
        at org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices$13.run(RMWebServices.java:2587)

Repro:

yarn-site.xml:

<property>
  <name>yarn.scheduler.configuration.fs.path</name> 
  <value>hdfs:///tmp/yarn/system</value>
</property>

 <property>
  <name>yarn.scheduler.configuration.store.class&lt;/name>
  <value>fs</value>
</property>

[yarn@yarndocker-1 yarn]$ cat /tmp/abc.xml 
<sched-conf>
        <update-queue>
          <queue-name>root.default</queue-name>
          <params>
            <entry>
              <key>priority</key>
              <value>10</value>
            </entry>
          </params>
        </update-queue>
      </sched-conf>

[yarn@yarndocker-1 yarn]$ curl -v -X PUT -d @/tmp/abc.xml -H "Content-type: application/xml"  'http://yarndocker-1:8088/ws/v1/cluster/scheduler-conf?user.name=yarn'
Filesystem closed

Issue Links

• links to: GitHub Pull Request #3262