Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-1150

Impala should limit the per-query core estimate to Yarn's per-node limit

    Details

      Description

      Yarn has yarn.nodemanager.resource.cpu-vcores which sets the maximum number of cores a single node can allocate. Impala uses Java's availableCores() method which looks at the whole machine. When RM is enabled, we should limit the estimated number of cores per-node to yarn.nodemanager.resource.cpu-vcores.

      This is probably easiest to do in the backend in QuerySchedule::GetPerHostVCores().

        Activity

        Hide
        henryr Henry Robinson added a comment -

        It's harder than I thought to get at the yarn conf without help from CM. There's a workaround by always using defaults for RM estimates.

        Show
        henryr Henry Robinson added a comment - It's harder than I thought to get at the yarn conf without help from CM. There's a workaround by always using defaults for RM estimates.
        Hide
        dhecht Dan Hecht added a comment -

        Not sure whether this will still be relevant in the new world.

        Show
        dhecht Dan Hecht added a comment - Not sure whether this will still be relevant in the new world.
        Hide
        mjacobs Matthew Jacobs added a comment -

        Llama/YARN code no longer exists

        Show
        mjacobs Matthew Jacobs added a comment - Llama/YARN code no longer exists

          People

          • Assignee:
            mjacobs Matthew Jacobs
            Reporter:
            henryr Henry Robinson
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development