[IMPALA-1852] Impala with RM/Llama acquires excessive thread tokens.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 1.4, Impala 2.0, Impala 2.1.1, Impala 2.2
• Fix Version/s: Impala 2.5.0
• Labels: llama performance resource-management
• Target Version: Impala 2.5.0
• Assignee: Matthew Jacobs
• Reporter: Alexander Behm
• Created: 06/Mar/15 02:28
• Updated: 23/Oct/15 21:29
• Resolved: 23/Oct/15 21:29

Description

In the attached profile there is a big discrepancy between "AverageThreadTokens" and the number of threads actually in use for a query. Acquiring so many thread tokens that are then not used can be a serious performance issue. In addition, this query tried to expand its vCore subscription to match the acquired thread tokens forever because Impala did not back off the expansion attempts even after all vCores for that node had been acquired.

Here's a relevant snippet from the profile:

      Instance 9d49cd02cdb3df07:4cf79895d9408ea8 (host=hadoop-slave08.local:22000):(Total: 164.143ms, non-child: 0ns, % non-child: 0.00%)
         - AverageThreadTokens: 95.88 <-- Excessive thread tokens!
         - PeakMemoryUsage: 29.38 GB (31550144512)
         - PerHostPeakMemUsage: 29.38 GB (31550144512)
         ...
        BlockMgr:
           ...  
        CodeGen:(Total: 400.797ms, non-child: 400.797ms, % non-child: 100.00%)
           ...
        DataStreamSender (dst_id=2):(Total: 19.128us, non-child: 19.128us, % non-child: 100.00%)
           ...
        AGGREGATION_NODE (id=1):(Total: 163.380ms, non-child: 0ns, % non-child: 0.00%)
            ...
        HDFS_SCAN_NODE (id=0):(Total: 10m9s, non-child: 10m9s, % non-child: 100.00%)
          Hdfs split stats (<volume id>:<# splits>/<split lengths>): 3:5/159.34 MB 2:8/254.94 MB 4:10/318.75 MB 1:9/286.93 MB 5:11/350.69 MB 0:8/255.09 MB 
          BytesRead(16s000ms): 926.64 MB, 956.53 MB, 956.53 MB, 956.53 MB, 956.53 MB, 956.53 MB, 956.53 MB, 956.53 MB, 956.53 MB, 956.53 MB, 956.53 MB, 956.53 MB, 956.53 MB, 956.53 MB, 956.53 MB, 956.53 MB, 956.53 MB, 956.53 MB, 956.53 MB, 956.53 MB, 956.53 MB, 956.53 MB, 956.53 MB, 956.53 MB, 956.53 MB, 956.53 MB, 956.53 MB, 956.53 MB, 956.53 MB, 956.53 MB, 956.53 MB, 956.53 MB, 956.53 MB, 956.53 MB, 956.53 MB, 956.53 MB, 956.53 MB, 956.53 MB
           ...
           - NumScannerThreadsStarted: 30 <-- Comparatively few threads
           - PeakMemoryUsage: 29.38 GB (31550115840)
           ...