[TEZ-4555] Fail fast in LocalClient if the dirs (log, local) haven't been created

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.10.4
• Assignee: László Bodor
• Reporter: László Bodor
• Created: 24/Apr/24 10:05
• Updated: 25/Apr/24 18:42
• Resolved: 25/Apr/24 10:20

Description

https://github.com/apache/tez/blob/f080031f5c72bc4bfd8090ccdc670bdc0f7fd090/tez-dag/src/main/java/org/apache/tez/client/LocalClient.java#L332-L335

          Path logDir = new Path(userDir, "localmode-log-dir");
          Path localDir = new Path(userDir, "localmode-local-dir");
          localFs.mkdirs(logDir);
          localFs.mkdirs(localDir);

in case of a non-writable local fs path (/base), this mkdirs silently returns with false, whereas I can see that it's not writable on my mac:

 mkdir -p /base
mkdir: /base: Read-only file system

leading to a confusing error message later:

2024-04-24T02:03:52,101 ERROR [DAGAppMaster Thread] client.LocalClient: Error starting DAGAppMaster
java.io.FileNotFoundException: /base/scratch/laszlobodor/_tez_session_dir/b76689bc-d25e-4d65-a339-44206ff57ce2/.tez/application_1713949431891_0001_wd/tez-conf.pb (No such file or directory)
	at java.io.FileInputStream.open0(Native Method) ~[?:1.8.0_292]
	at java.io.FileInputStream.open(FileInputStream.java:195) ~[?:1.8.0_292]
	at java.io.FileInputStream.<init>(FileInputStream.java:138) ~[?:1.8.0_292]
	at org.apache.tez.common.TezUtilsInternal.readUserSpecifiedTezConfiguration(TezUtilsInternal.java:84) ~[tez-common-0.9.1.2024.0.19.0-3.jar:0.9.1.2024.0.19.0-3]
	at org.apache.tez.client.LocalClient.createDAGAppMaster(LocalClient.java:394) ~[tez-dag-0.9.1.2024.0.19.0-3.jar:0.9.1.2024.0.19.0-3]
	at org.apache.tez.client.LocalClient$1.run(LocalClient.java:357) [tez-dag-0.9.1.2024.0.19.0-3.jar:0.9.1.2024.0.19.0-3]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292]

actually, the fix should be done in HIVE-28212, but we need to fail fast here and give a hint to the user about the folder

Issue Links

• relates to: HIVE-28212 MiniHS2: use a base folder which is more likely writable on the local FS
• links to: GitHub Pull Request #348