I run a spark job on an EMR cluster when using the SparkSession.Builder to create a spark Session.
The job includes querying a Hive table, and when it is executed Spark throws "
org.apache.spark.sql.AnalysisException: Table or view not found".
I printed all the config options in Spark context and noticed that "spark.sql.catalogImplementation" is missing.
If I provide this option as a "-conf" in "spark-submit" then it is added and everything works.
This is the code I use to instantiate the SparkSession:
val spark: SparkSession = SparkSession
Also note that I added the following two debugging statements just before the code that creates the session to see if there was already a session in place, and they both returned "false":