[FLINK-34568] YarnFileStageTest.destroyHDFS timed out

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 1.17.2
• Component/s: Connectors / Hadoop Compatibility
• Labels: test-stability
• Assignee: Unassigned
• Reporter: Matthias Pohl
• Created: 04/Mar/24 07:24
• Updated: 04/Mar/24 07:24

Description

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58024&view=logs&j=5cae8624-c7eb-5c51-92d3-4d2dacedd221&t=5acec1b4-945b-59ca-34f8-168928ce5199&l=26698

Mar 02 07:28:56 "Listener at localhost/33933" #25 daemon prio=5 os_prio=0 tid=0x00007f08490be000 nid=0x12cae runnable [0x00007f082ebfc000]
Mar 02 07:28:56    java.lang.Thread.State: RUNNABLE
Mar 02 07:28:56 	at org.mortbay.io.nio.SelectorManager$SelectSet.stop(SelectorManager.java:879)
Mar 02 07:28:56 	- locked <0x00000000d7ae0030> (a org.mortbay.io.nio.SelectorManager$SelectSet)
[...]
Mar 02 07:28:56 	at org.apache.hadoop.hdfs.MiniDFSCluster.stopAndJoinNameNode(MiniDFSCluster.java:2123)
Mar 02 07:28:56 	at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:2060)
Mar 02 07:28:56 	at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:2031)
Mar 02 07:28:56 	at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:2024)
Mar 02 07:28:56 	at org.apache.flink.yarn.YarnFileStageTest.destroyHDFS(YarnFileStageTest.java:90)
[...]

Looks like a HDFS issue during shutdown? This will most likely also affect newer versions because there was not much done in the Yarn space since 1.17 (hadoop was bumped in 1.17 itself; FLINK-29710).