Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-9665

Parallel move task optimization causes race condition

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.2.0
    • Component/s: None
    • Labels:
      None

      Description

      The change in HIVE-8042 doesn't actually work. Running it at scale produces race conditions which lead to broken thrift messages and OOMs. E.g.:

      java.lang.OutOfMemoryError: Java heap space
      	at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:353)
      	at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:215)
      	at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
      	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table(ThriftHiveMetastore.java:1122)
      	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table(ThriftHiveMetastore.java:1108)
      	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1091)
      	at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.getTable(SessionHiveMetaStoreClient.java:131)
      	at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:90)
      	at com.sun.proxy.$Proxy9.getTable(Unknown Source)
      	at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1064)
      	at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1019)
      	at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1006)
      	at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:250)
      	at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
      	at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
      	at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:72)
      java.lang.OutOfMemoryError: Java heap space
      	at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:353)
      	at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:215)
      	at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
      	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table(ThriftHiveMetastore.java:1122)
      	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table(ThriftHiveMetastore.java:1108)
      	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1091)
      	at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.getTable(SessionHiveMetaStoreClient.java:131)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:90)
      	at com.sun.proxy.$Proxy9.getTable(Unknown Source)
      	at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1064)
      	at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1019)
      	at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1006)
      	at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:250)
      	at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
      	at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
      	at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:72)
      java.lang.OutOfMemoryError: Java heap space
      	at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:353)
      	at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:215)
      	at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
      	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table(ThriftHiveMetastore.java:1122)
      	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table(ThriftHiveMetastore.java:1108)
      	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1091)
      	at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.getTable(SessionHiveMetaStoreClient.java:131)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:90)
      	at com.sun.proxy.$Proxy9.getTable(Unknown Source)
      	at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1064)
      	at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1019)
      	at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1006)
      	at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:250)
      	at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
      	at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
      	at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:72)
      

        Attachments

        1. HIVE-9665.1.patch
          1 kB
          Gunther Hagleitner

          Issue Links

            Activity

              People

              • Assignee:
                hagleitn Gunther Hagleitner
                Reporter:
                hagleitn Gunther Hagleitner
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: