Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4064

impala query is spent long time for ScannerThreadsTotalWallClockTime

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Information Provided
    • Affects Version/s: Impala 2.3.0
    • Fix Version/s: None
    • Component/s: Perf Investigation
    • Labels:
    • Environment:
      CDH 5.7.2-impala 2.3

      Description

      In our CDH 5.5.2 with impala 2.3, we often met performance issue, the same queries in the same cluster, sometimes is fast, some times is very slow, our sql is very simple with only one aggregation(for detail pls check the query profile info) like bellow:
      ```sql
      select Sum(c1), sum(c2)
      FROM xxx
      WHERE day between 20160818 and 20160824
      ```

      we also check the profile, compare the slow and quick for the same sql, we found that: the slow query is wasting time for ScannerThreadsTotalWallClockTime, and only one node is extremely slow, others are quick. we also found the spent time is mostly wasted in InactiveTotalTime, it seems the cost is wasted on hdfs scan, but the all other executor node is not met the problem, we also checked the namenode server resource(no resource tension, include memory,cpu,network), the attached is the quick and slow profile data for the same query, the query only used less than 8M memory for each executor.

      pls help us, tell me what exactly cost for ScannerThreadsTotalWallClockTime, how can we improve it? how can we make query stable and quickly, thx very much.

        Attachments

        1. ScannerThreadsTotalWallClockTime.png
          73 kB
          hongchao
        2. inactivetotaltime.png
          20 kB
          hongchao
        3. profile_quick.txt
          76 kB
          hongchao
        4. profile_slow.txt
          76 kB
          hongchao

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              hongchao_impala_680d hongchao
            • Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: