Details
-
Bug
-
Status: Open
-
Critical
-
Resolution: Unresolved
-
2.3.2
-
None
-
None
Description
Hive queries are stuck after initializing MapOperator. These hive queries are simple CTAS reading from a Hive table backed by rcfile format. This table has 7500 partitions and 110 columns with column data types restricted to string and int. This is being on run on a EMR cluster with 100 data nodes with enough memory/cores.
After the query is submitted, YARN allocates the necessary containers. All the mapper tasks are in RUNNING state and all the map tasks reach the stage of initializing MapOperator and get stuck. Here is log message from the map tasks.
2019-01-17 15:02:06,262 INFO [main] org.apache.hadoop.hive.ql.exec.MapOperator: Initializing operator MAP[0]
2019-01-17 15:08:22,093 INFO [main] org.apache.hadoop.hive.ql.exec.mr.ExecMapper:
You can see that task is stuck for 6 minutes. This even gets to 20 minutes depending upon the number of parallel queries on the cluster. After the ExceMapper starts, the query completes in a minute.
I also noticed tread dumps in the logs that lead to a finding where it is spending all the time in this menthod setReadNestedColumnPathConf().
"main" #1 prio=5 os_prio=0 tid=0x00007f4cd805e800 nid=0x18074 runnable [0x00007f4cded7f000]
java.lang.Thread.State: RUNNABLE
at java.lang.String.toLowerCase(String.java:2670)
at org.apache.hadoop.hive.serde2.ColumnProjectionUtils.setReadNestedColumnPathConf(ColumnProjectionUtils.java:223)
at org.apache.hadoop.hive.serde2.ColumnProjectionUtils.appendNestedColumnPaths(ColumnProjectionUtils.java:145)
at org.apache.hadoop.hive.ql.exec.MapOperator.cloneConfsForNestedColPruning(MapOperator.java:365)
at org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:419)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:106)
Based on the error, I tried running the same query with 1 to 5 columns in the select clause and it runs quickly as expected. If I include more columns in the select clause, it falls in to the same issue with long pause times between MapOperator initialization and ExecMapper.
The error is very similar to the one in this Jira
https://issues.apache.org/jira/browse/HIVE-16969