Major Description
The current SessionState initialization path is quite complex. A part of the creation is done in the SessionState companion objects, a part of the creation is one inside the SessionState class, and a part is done by passing functions.
The proposal is to consolidate the SessionState initialization into a builder class. This SessionState will not do any initialization and just becomes a place holder for the various Spark SQL internals. The advantages of this approach are the following:
- SessionState initialization is less dispersed. The builder should be a one stop shop.
- This provides us with a start for removing the HiveSessionState. Removing the hive session state would also require us to move resource loading into a separate class, and to (re)move metadata hive.
- It is easier to customize the Spark Session. You just need to create a custom version of the builder. I will add hooks to make this easier. Opening up these API's will happen at a later point.
