Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-5605

document how to increase thread resource limits

    XMLWordPrintableJSON

    Details

    • Type: Task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: Impala 2.9.0
    • Fix Version/s: Impala 3.2.0
    • Component/s: Docs
    • Labels:
      None

      Description

      Depending on the workload, Impala may need to create a very large number of threads. If so, it is necessary to configure the system correctly to prevent Impala from crashing because of resource limitations. Such a crash would look like this:

      F0629 08:20:02.956413 29088 llvm-codegen.cc:111] LLVM hit fatal error: Unable to allocate section memory!
      terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::thread_resource_error> >'

      To prevent this, each Impala host should be configured like this:

      echo 2000000 > /proc/sys/kernel/threads-max
      echo 2000000 > /proc/sys/kernel/pid_max
      echo 8000000 > /proc/sys/vm/max_map_count

      In /etc/security/limits.conf add

      impala soft nproc 262144
      impala hard nproc 262144

        Attachments

          Activity

            People

            • Assignee:
              arodoni Alexandra Rodoni
              Reporter:
              mmulder Matthew Mulder
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: