[HIVE-21447] Constraints: Adding a constraint twice takes 120+ seconds to fail

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 4.0.0
• Component/s: Metastore, Standalone Metastore
• Assignee: Unassigned
• Reporter: Gopal Vijayaraghavan
• Created: 14/Mar/19 06:14
• Updated: 14/Mar/19 06:14

Description

It reconnects to the metastore 24 times before failing without a real reason.

There's no useful error returned.

 alter table customer add constraint ${DB}_c_fsd foreign key  (c_first_sales_date_sk) references date_dim (d_date_sk) disable novalidate rely;
INFO  : Compiling command(queryId=hive_20190314061155_a060ca11-9547-4400-9564-9f0108c0e530): alter table customer add constraint tpcds_copy_orc_partitioned_10000_c_fsd foreign key  (c_first_sales_date_sk) references date_dim (d_date_sk) disable novalidate rely
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20190314061155_a060ca11-9547-4400-9564-9f0108c0e530); Time taken: 0.021 seconds
INFO  : Executing command(queryId=hive_20190314061155_a060ca11-9547-4400-9564-9f0108c0e530): alter table customer add constraint tpcds_copy_orc_partitioned_10000_c_fsd foreign key  (c_first_sales_date_sk) references date_dim (d_date_sk) disable novalidate rely
INFO  : Starting task [Stage-0:DDL] in serial mode
ERROR : FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. org.apache.thrift.TApplicationException: Internal error processing add_foreign_key
INFO  : Completed executing command(queryId=hive_20190314061155_a060ca11-9547-4400-9564-9f0108c0e530); Time taken: 122.712 seconds
Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. org.apache.thrift.TApplicationException: Internal error processing add_foreign_key (state=08S01,code=1)

2019-03-14T06:12:57,299 WARN  [HiveServer2-Background-Pool: Thread-4295 (hive_20190314061155_a060ca11-9547-4400-9564-9f0108c0e530)] metastore.RetryingMetaStoreClient: MetaStoreClientlost connection. Attempting to reconnect (13 of 24) after 5s. addForeignKey
org.apache.thrift.TApplicationException: Internal error processing add_foreign_key
        at org.apache.thrift.TApplicationException.read(TApplicationException.java:111) ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
        at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79) ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_add_foreign_key(ThriftHiveMetastore.java:1738) ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.add_foreign_key(ThriftHiveMetastore.java:1725) ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.addForeignKey(HiveMetaStoreClient.java:1033) ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
        at sun.reflect.GeneratedMethodAccessor323.invoke(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_181]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181]
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212) ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
        at com.sun.proxy.$Proxy36.addForeignKey(Unknown Source) ~[?:?]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_181]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_181]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_181]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181]
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:3010) ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
        at com.sun.proxy.$Proxy36.addForeignKey(Unknown Source) ~[?:?]
        at org.apache.hadoop.hive.ql.metadata.Hive.addForeignKey(Hive.java:5504) ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.exec.DDLTask.addConstraints(DDLTask.java:4080) ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]