[HIVE-1219] More robust handling of metastore connection failures

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.6.0
• Component/s: Metastore
• Hadoop Flags: Reviewed
• Assignee: Paul Yang
• Reporter: Paul Yang
• Created: 10/Mar/10 03:35
• Updated: 17/Dec/11 00:03
• Resolved: 24/Mar/10 21:17

Description

Currently, if metastore's connection to the datastore is broken, the query fails and the exception such as the following is thrown

2010-01-28 11:50:20,885 ERROR exec.MoveTask (SessionState.java:printError(248)) - Failed with exception Unable to fetch table tmp_table
org.apache.hadoop.hive.ql.metadata.HiveException: Unable to fetch table tmp_table
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:362)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:333)
at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:112)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:99)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:64)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:582)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:462)
at org.apache.hadoop.hive.ql.Driver.runCommand(Driver.java:324)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:312)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:123)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:181)
at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:200)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:256)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: javax.jdo.JDODataStoreException: Communications link failure

Last packet sent to the server was 1 ms ago.
NestedThrowables:
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

Last packet sent to the server was 1 ms ago.
at org.datanucleus.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:289)

In order to reduce the impact of transient network issues and momentarily unavailable datastores, two possible improvements are:

1. Retrying the metastore command in case of connection failure before propagating up the exception.

2. Retrieving the datastore hostname / connection URL through the use of an extension. This extension would be useful in the case where a remote service maintained the location of the currently available datastore. In case of hostname changes or failovers to a backup datastore, the extension would allow hive clients to run without manual intervention.

Issue Links

• relates to: HIVE-2029 MetaStore ConnectionURL updates need to trigger creation of Default DB if it doesn't exist