Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4522

Impalad crash under stress due to too many Kudu client threads

    Details

      Description

      In stress testing on physical boxes (80 cores, 200gb ram) we discovered that the Kudu Java client creates a huge number of threads (2x the #cores) per Kudu client. Impala creates a Kudu client in catalog (table loading and DDL), during planning (fetching tablet locations for scans), and in the BE. The client created in planning is particularly problematic wrt the number of threads created, especially on machines with a large number of cores and under load. In the stress tests, this could result in the process crashing at some point when the JVM could not create more threads.

      The attached hs_err.log shows this crash.

      While we should explore sharing a single Kudu client (1 client needed per master), in the meantime we should reduce the number of threads the client creates, which is exposed in the Kudu API.

        Attachments

        1. hs_err.log
          3.47 MB
          Matthew Jacobs

          Activity

            People

            • Assignee:
              mjacobs Matthew Jacobs
              Reporter:
              mjacobs Matthew Jacobs
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: