Details
Major Description
Due to the recent tough problems encountered in impala, we may decide to abandon its use in production, This is a great pity. !!! We have used impala for several years and have tried to solve some problems, but this time we have tried a lot of trials and it has no effect.
The system applies impala to ETL data processing. Currently, the main problem is that the SQL task startup delay submitted by the client is in the CREATED state on the Coordinator 25000 web, not the Running state. These tasks cannot be seen quickly on the CM interface.
.It is possible to wait for a while to run past, or it may stay stuck and die for a long time. The situation may seem to be related to metadata loading, related to the catalogd service. We have more than 300,000 tables and 8 million partitions, and have tried to reduce some useless tables, but in the end did not capture slow queries on the Mysql metabase. we've tried to restart Hive ,Namenode.
Later, we found the problem was similar to that described by IMPALA-5058. We tried to upgrade the version of impala to 2.12.0 using overwritten file mode. The service log became more detailed, but it was still not solved. Tracking the log of catalogd found the following rule. Once "Remaining items in queue: 0, Loads in progress: 1" appears in some worker threads, the problem will appear. We tried to analyze the jstack information of catalogd and found that a large number of threads are in Waiting state.
This is our cluster environment:
CDH 5.13.1 impala-2.10.0 & 2.12.0
300 datanodes, 80 impalad
