Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-9777

YARN: JM and TM Memory must be specified with Units

    XMLWordPrintableJSON

    Details

      Description

      FLINK-6469 breaks backwards compatibility because the JobManager and TaskManager memory must be specified with units (otherwise bytes are assumed). The command to start a YARN session as documented (https://github.com/apache/flink/blob/9f736d1927c62d220a82931c4f5ffa4955910f27/docs/ops/deployment/yarn_setup.md) would not work because 1024 bytes and 4096 bytes are not enough for the heap size. The command finishes with the following exception:

      java.lang.reflect.UndeclaredThrowableException
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1854)
      	at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
      	at org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:802)
      Caused by: org.apache.flink.client.deployment.ClusterDeploymentException: Couldn't deploy Yarn session cluster
      	at org.apache.flink.yarn.AbstractYarnClusterDescriptor.deploySessionCluster(AbstractYarnClusterDescriptor.java:420)
      	at org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:599)
      	at org.apache.flink.yarn.cli.FlinkYarnSessionCli.lambda$main$2(FlinkYarnSessionCli.java:802)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:422)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
      	... 2 more
      Caused by: org.apache.flink.util.FlinkException: Cannot fulfill the minimum memory requirements with the provided cluster specification. Please increase the memory of the cluster.
      	at org.apache.flink.yarn.AbstractYarnClusterDescriptor.validateClusterSpecification(AbstractYarnClusterDescriptor.java:453)
      	at org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:475)
      	at org.apache.flink.yarn.AbstractYarnClusterDescriptor.deploySessionCluster(AbstractYarnClusterDescriptor.java:413)
      	... 7 more
      Caused by: java.lang.IllegalArgumentException
      	at org.apache.flink.util.Preconditions.checkArgument(Preconditions.java:123)
      	at org.apache.flink.runtime.clusterframework.ContaineredTaskManagerParameters.calculateCutoffMB(ContaineredTaskManagerParameters.java:115)
      	at org.apache.flink.yarn.AbstractYarnClusterDescriptor.validateClusterSpecification(AbstractYarnClusterDescriptor.java:450)
      	... 9 more
      

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                yanghua vinoyang
                Reporter:
                gjy Gary Yao
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: