Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-1219

More robust handling of metastore connection failures

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.6.0
    • Component/s: Metastore
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently, if metastore's connection to the datastore is broken, the query fails and the exception such as the following is thrown

      2010-01-28 11:50:20,885 ERROR exec.MoveTask (SessionState.java:printError(248)) - Failed with exception Unable to fetch table tmp_table
      org.apache.hadoop.hive.ql.metadata.HiveException: Unable to fetch table tmp_table
      at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:362)
      at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:333)
      at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:112)
      at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:99)
      at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:64)
      at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:582)
      at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:462)
      at org.apache.hadoop.hive.ql.Driver.runCommand(Driver.java:324)
      at org.apache.hadoop.hive.ql.Driver.run(Driver.java:312)
      at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:123)
      at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:181)
      at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:200)
      at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:256)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
      Caused by: javax.jdo.JDODataStoreException: Communications link failure
      
      Last packet sent to the server was 1 ms ago.
      NestedThrowables:
      com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
      
      Last packet sent to the server was 1 ms ago.
      at org.datanucleus.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:289)
      

      In order to reduce the impact of transient network issues and momentarily unavailable datastores, two possible improvements are:

      1. Retrying the metastore command in case of connection failure before propagating up the exception.

      2. Retrieving the datastore hostname / connection URL through the use of an extension. This extension would be useful in the case where a remote service maintained the location of the currently available datastore. In case of hostname changes or failovers to a backup datastore, the extension would allow hive clients to run without manual intervention.

        Attachments

        1. HIVE-1219.9.patch
          65 kB
          Paul Yang
        2. HIVE-1219.8.patch
          64 kB
          Paul Yang
        3. HIVE-1219.7.patch
          64 kB
          Paul Yang
        4. HIVE-1219.6.patch
          61 kB
          Paul Yang
        5. HIVE-1219.5.patch
          58 kB
          Paul Yang
        6. HIVE-1219.4.patch
          63 kB
          Paul Yang

          Issue Links

            Activity

              People

              • Assignee:
                pauly Paul Yang
                Reporter:
                pauly Paul Yang
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: