[HIVE-18986] Table rename will run java.lang.StackOverflowError in dataNucleus if the table contains large number of columns

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0-alpha-1
• Component/s: Standalone Metastore
• Release Note: Added HMS configuration {{metastore.rawstore.batch.size}} to control batch size when executing the query using JDO. That will avoid the issues like StackOverflow if lots of objects such as column names are involved in the query.
• Assignee: Aihua Xu
• Reporter: Aihua Xu
• Created: 17/Mar/18 01:17
• Updated: 17/Nov/22 08:54
• Resolved: 25/Apr/18 23:15

Description

If the table contains a lot of columns e.g, 5k, simple table rename would fail with the following stack trace. The issue is datanucleus can't handle the query with lots of colName='c1' && colName='c2' && ... .

 

2018-03-13 17:19:52,770 INFO org.apache.hadoop.hive.metastore.HiveMetaStore.audit: [pool-5-thread-200]: ugi=anonymous ip=10.17.100.135 cmd=source:10.17.100.135 alter_table: db=default tbl=fgv_full_var_pivoted02 newtbl=fgv_full_var_pivoted 2018-03-13 17:20:00,495 ERROR org.apache.hadoop.hive.metastore.RetryingHMSHandler: [pool-5-thread-200]: java.lang.StackOverflowError at org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:330) at org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339) at org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339) at org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339) at org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339)

 

Issue Links

• links to: code review