Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-7224

Could not find or load main class org.apache.hadoop.yarn.server.nodemanager.containermanager.loca lizer.ContainerLocalizer when yarn on kerberos

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.1.0, 3.2.0
    • Fix Version/s: None
    • Component/s: job submission
    • Labels:
    • Environment:

      The environment:

      CentOS Linux release 7.5.1804 (Core) ;

      java version "1.8.0_171" ;

      Hadoop 3.1.0 ,3.2.0 ;

      kerberos 1.15.1-34 ;

       

      Description

       

      cat /etc/container-executor.cfg

      yarn.nodemanager.linux-container-executor.group=hadoop#
      banned.users=#comma separated list of users who can not run applications
      min.user.id=1#Prevent other super-users
      allowed.system.users=jingwei.shi#

       

      cat  yarn-site.xml

      <property>
      <name>yarn.resourcemanager.keytab</name>
      <value>/var/kerberos/krb5/user/hadoop.keytab</value>
      </property>

      <property>
      <name>yarn.resourcemanager.principal</name>
      <value>hadoop/_HOST@HADOOP.COM</value>
      </property>

      <property>
      <name>yarn.nodemanager.keytab</name>
      <value>/var/kerberos/krb5/user/hadoop.keytab</value>
      </property>

      <property>
      <name>yarn.nodemanager.principal</name>
      <value>hadoop/_HOST@HADOOP.COM</value>
      </property>

      <property>
      <name>yarn.nodemanager.default-container-executor.log-dirs.permissions</name>
      <value>777</value>
      </property>

      <property>
      <name>yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users</name>
      <value>false</value>
      </property>

      <property>
      <name>yarn.nodemanager.container-executor.class</name>
      <value>org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor</value>
      <!-value>org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor</value->
      </property>

      <property>
      <name>yarn.nodemanager.linux-container-executor.group</name>
      <value>hadoop</value>
      </property>

      <property>
      <name>yarn.nodemanager.local-dirs</name>
      <value>/tmp/userlogs</value>
      </property>

      <property>
      <name>yarn.nodemanager.log-dirs</name>
      <value>/tmp/userlogs</value>
      </property>

      ++++++++++++++++++++++++++++++++++

      klist
      Ticket cache: FILE:/tmp/krb5cc_1000
      Default principal: hadoop/bj-jd-dc-namenode-prod-0003.tendcloud.com@HADOOP.COM

      Valid starting Expires Service principal
      07/12/2019 16:05:00 07/13/2019 16:05:00 krbtgt/HADOOP.COM@HADOOP.COM
      renew until 07/26/2019 16:05:00

      Hadoop user submitted tasks successfully ;

      ————————————————————————————————

      while other user submitted tasks showed that no classes were found .

      klist
      Ticket cache: FILE:/tmp/krb5cc_1000
      Default principal: jingwei.shi/datanode-prod-0011.tendcloud.com@HADOOP.COM

      Valid starting Expires Service principal
      07/12/2019 14:50:37 07/13/2019 14:50:24 krbtgt/HADOOP.COM@HADOOP.COM
      renew until 07/26/2019 14:50:24

      2019-07-12 15:28:01,225 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed for container_e28_1562916339074_0001_02_000001
      java.io.IOException: Application application_1562916339074_0001 initialization failed (exitCode=1) with output: main : command provided 0
      main : run as user is jingwei.shi
      main : requested yarn user is jingwei.shi

      at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.startLocalizer(LinuxContainerExecutor.java:411)
      at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1245)
      Caused by: org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationException: ExitCodeException exitCode=1: Error: Could not find or load main class org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              hrsjw1 jingwei.shi
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: