[FLINK-30793] PyFlink YARN per-job on Docker test fails on Azure due to permission issues

• Type: Bug
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 1.17.0
• Component/s: Connectors / Hadoop Compatibility, Deployment / YARN
• Labels: auto-deprioritized-major test-stability
• Assignee: Unassigned
• Reporter: Matthias Pohl
• Created: 26/Jan/23 07:50
• Updated: 04/Apr/24 15:59

Description

The following build failed due to some hdfs/yarn permission issues in PyFlink YARN per-job on Docker e2e test:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=45202&view=logs&j=af184cdd-c6d8-5084-0b69-7e9c67b35f7a&t=160c9ae5-96fd-516e-1c91-deb81f59292a&l=10587

[...]
Jan 26 02:17:31 23/01/26 02:12:20 FATAL hs.JobHistoryServer: Error starting JobHistoryServer
Jan 26 02:17:31 org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Error creating done directory: [hdfs://master.docker-hadoop-cluster-network:9000/tmp/hadoop-yarn/staging/history/done]
Jan 26 02:17:31 	at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.tryCreatingHistoryDirs(HistoryFileManager.java:698)
Jan 26 02:17:31 	at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.createHistoryDirs(HistoryFileManager.java:634)
Jan 26 02:17:31 	at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.serviceInit(HistoryFileManager.java:595)
Jan 26 02:17:31 	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
Jan 26 02:17:31 	at org.apache.hadoop.mapreduce.v2.hs.JobHistory.serviceInit(JobHistory.java:96)
Jan 26 02:17:31 	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
Jan 26 02:17:31 	at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
Jan 26 02:17:31 	at org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.serviceInit(JobHistoryServer.java:152)
Jan 26 02:17:31 	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
Jan 26 02:17:31 	at org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.launchJobHistoryServer(JobHistoryServer.java:228)
Jan 26 02:17:31 	at org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.main(JobHistoryServer.java:238)
Jan 26 02:17:31 Caused by: org.apache.hadoop.security.AccessControlException: Permission denied: user=mapred, access=WRITE, inode="/":hdfs:hadoop:drwxr-xr-x
Jan 26 02:17:31 	at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:350)
Jan 26 02:17:31 	at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:251)
Jan 26 02:17:31 	at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:189)
Jan 26 02:17:31 	at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1756)
Jan 26 02:17:31 	at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1740)
Jan 26 02:17:31 	at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:1699)
Jan 26 02:17:31 	at org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:60)
Jan 26 02:17:31 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3007)
Jan 26 02:17:31 	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:1141)
Jan 26 02:17:31 	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:659)
Jan 26 02:17:31 	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
[...]

Issue Links

• is related to: FLINK-24434 PyFlink YARN per-job on Docker test fails on Azure