[HUDI-2747] Fix hudi cli metadata commands

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.10.0
• Fix Version/s: 0.11.0
• Labels: HUDI-bug pull-request-available
• Story Points: 1
• Epic Link: Metadata Table for File Listing & Query Planning
• Assignee: Yue Zhang
• Reporter: sivabalan narayanan

Reviewers

Alexey Kudinkin

• Created: 12/Nov/21 03:13
• Updated: 08/Mar/22 04:06
• Resolved: 08/Mar/22 04:06

Description

Fix hudi cli metadata commands.  Currently when running hudi cli metadata commands locally, it complains about spark master, e.g., yarn master not found.  The commands need to take in spark related configs so that they can run locally and in cluster.

java.lang.reflect.UndeclaredThrowableException
	at org.springframework.util.ReflectionUtils.rethrowRuntimeException(ReflectionUtils.java:315)
	at org.springframework.util.ReflectionUtils.handleInvocationTargetException(ReflectionUtils.java:295)
	at org.springframework.util.ReflectionUtils.handleReflectionException(ReflectionUtils.java:279)
	at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:219)
	at org.springframework.shell.core.SimpleExecutionStrategy.invoke(SimpleExecutionStrategy.java:68)
	at org.springframework.shell.core.SimpleExecutionStrategy.execute(SimpleExecutionStrategy.java:59)
	at org.springframework.shell.core.AbstractShell.executeCommand(AbstractShell.java:134)
	at org.springframework.shell.core.JLineShell.promptLoop(JLineShell.java:533)
	at org.springframework.shell.core.JLineShell.run(JLineShell.java:179)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.spark.SparkException: Could not parse Master URL: 'yarn'
	at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2784)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:493)
	at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
	at org.apache.hudi.cli.utils.SparkUtil.initJavaSparkConf(SparkUtil.java:117)
	at org.apache.hudi.cli.commands.MetadataCommand.initJavaSparkContext(MetadataCommand.java:362)
	at org.apache.hudi.cli.commands.MetadataCommand.listPartitions(MetadataCommand.java:196)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:216)
	... 6 more 

Issue Links

• links to: GitHub Pull Request #4964