When there are multiple requests to the catalogd to prioritize loading the same table, then several catalog loading threads may end up waiting for that single table to be loaded, effectively reducing the number of catalog loading threads. In extreme examples, this might degrade to serial loading of tables.
Note that even a single query may issue multiple table-loading requests even for the same table if the table is very big. After issuing a load request, an impalad will wait 2m for the metadata to arrive, and then send the request again every 2m. So if a large table takes 20m to load, then a single query could issue 10 table-loading requests which ultimately hog 10 table-loading threads in the catalogd.
The simplest way to diagnose the issue is to examine the jstack of the catalogd and then you might discover several stacks that look like this:
The buggy code can be found in TableLoadingMgr.java:
Notice that the first few lines are intended to avoid loading the same table multiple times. However, the code does not prevent multiple threads from entering Catalog.getTableOrLoad() which will block on the same future for the same table.
The issue is easy to reproduce by simulating a long table load and doing several concurrent loads of the same table from an impalad. For example, you can first "invalidate metadata t" and then "desc t" several times concurrently.
A slow table loading can be simulated by adding a sleep inside call() function of the FutureTask created in TableLoadingMgr.loadAsync().