[SEDONA-534] Disable Python warning message of finding jars

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.5.2
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: Jia Yu
• Created: 19/Apr/24 05:29
• Updated: 28/Apr/24 08:22
• Resolved: 28/Apr/24 08:22

Description

Sedona Python tries to find what jars are used. However, if a user puts a jar in SPARK_HOME/jars directly, this will give annoying warning message

 

```
WARNING root:config.py:160 Failed to get the value of spark.jars from SparkConf: An error occurred while calling o132.get.
443: java.util.NoSuchElementException: spark.jars
444 at org.apache.spark.sql.errors.QueryExecutionErrors$.noSuchElementExceptionError(QueryExecutionErrors.scala:1660)
445 at org.apache.spark.sql.internal.SQLConf.$anonfun$getConfString$3(SQLConf.scala:4546)
446 at scala.Option.getOrElse(Option.scala:189)
447 at org.apache.spark.sql.internal.SQLConf.getConfString(SQLConf.scala:4546)
448 at org.apache.spark.sql.RuntimeConfig.get(RuntimeConfig.scala:72)
449 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
450 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
451 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
452 at java.lang.reflect.Method.invoke(Method.java:498)
453 at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
454 at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
455 at py4j.Gateway.invoke(Gateway.java:282)
456 at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
457 at py4j.commands.CallCommand.execute(CallCommand.java:79)
458 at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
459 at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
460 at java.lang.Thread.run(Thread.java:750)
461
462WARNING root:config.py:160 Failed to get the value of spark.yarn.dist.jars from SparkConf: An error occurred while calling o132.get.
463: java.util.NoSuchElementException: spark.yarn.dist.jars
464 at org.apache.spark.sql.errors.QueryExecutionErrors$.noSuchElementExceptionError(QueryExecutionErrors.scala:1660)
465 at org.apache.spark.sql.internal.SQLConf.$anonfun$getConfString$3(SQLConf.scala:4546)
466 at scala.Option.getOrElse(Option.scala:189)
467 at org.apache.spark.sql.internal.SQLConf.getConfString(SQLConf.scala:4546)
468 at org.apache.spark.sql.RuntimeConfig.get(RuntimeConfig.scala:72)
469 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
470 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
471 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
472 at java.lang.reflect.Method.invoke(Method.java:498)
473 at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
474 at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
475 at py4j.Gateway.invoke(Gateway.java:282)
476 at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
477 at py4j.commands.CallCommand.execute(CallCommand.java:79)
478 at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
479 at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
480 at java.lang.Thread.run(Thread.java:750)
```
 

Issue Links

• links to: GitHub Pull Request #1344