Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-3787

Yarn client does not report unfulfillable container constraints

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Duplicate
    • Affects Version/s: 1.1.0
    • Fix Version/s: 1.2.0, 1.1.4
    • Component/s: YARN
    • Labels:
      None

      Description

      If the number of virtual cores for a Yarn container is not fulfillable, then the TaskManager won't be started. This is only reported in the logs but not in the FlinkYarnClient. Thus, the user will see a started JobManager with no connected TaskManagers. Since the log aggregation is only available after the Yarn job has been stopped, there is no easy way for the user to detect what's going on.

      This problem is aggravated by the fact that the number of virtual cores is coupled to the number of slots if no explicit value has been set for the virtual cores. Therefore, it might happen that the Yarn deployment fails because of the virtual cores even though the user has never set a value for them (the user might even not know about the virtual cores).

      I think it would be good to check if the virtual cores constraint is fulfillable. If not, then the user should receive a clear message that the Flink cluster cannot be deployed (similar to the memory constraints).

        Issue Links

          Activity

          Hide
          mxm Maximilian Michels added a comment -

          This has been fixed in the meantime in FLINK-4998.

          Show
          mxm Maximilian Michels added a comment - This has been fixed in the meantime in FLINK-4998 .
          Hide
          mxm Maximilian Michels added a comment -

          Do we want to fix this for the 1.1.0 release?

          Show
          mxm Maximilian Michels added a comment - Do we want to fix this for the 1.1.0 release?

            People

            • Assignee:
              roman_maier Roman Maier
              Reporter:
              till.rohrmann Till Rohrmann
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development