[SPARK-4758] Make metastore_db in-memory for HiveContext

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Won't Fix
• Affects Version/s: 1.2.0, 1.3.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Jianshi Huang
• Created: 05/Dec/14 06:27
• Updated: 15/Sep/15 21:15
• Resolved: 15/Sep/15 21:15

Description

HiveContext by default will create a local folder metastore_db.

This is not very user friendly as the metastore_db will be locked by HiveContext and thus will block multiple Spark process to start from the same directory.

I would propose adding a default hive-site.xml in conf/ with the following content.

<configuration>

<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:derby:memory:databaseName=metastore_db;create=true</value>
</property>

<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>org.apache.derby.jdbc.EmbeddedDriver</value>
</property>

<property>
<name>hive.metastore.warehouse.dir</name>
<value>file://${user.dir}/hive/warehouse</value>
</property>

</configuration>

jdbc:derby:memory:databaseName=metastore_db;create=true Will make sure the embedded derby database is created in-memory.

Jianshi