Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-7054 Yarn Service Phase 2
  3. YARN-7543

FileNotFoundException due to a broken link when creating a yarn service and missing max cpu limit check

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: yarn-native-services
    • Component/s: None
    • Labels:
      None

      Description

      The hadoop lib dir had a broken link to a ojdb jar which was not really required for a YARN service creation. The app submission failed with the below FNFE. Ideally it should be handled and app should be successfully submitted and let the app fail if it really needed the jar of the broken link -

      [root@ctr-e134-1499953498516-324910-01-000002 ~]# yarn app -launch gour-sleeper sleeper
      WARNING: YARN_LOG_DIR has been replaced by HADOOP_LOG_DIR. Using value of YARN_LOG_DIR.
      WARNING: YARN_LOGFILE has been replaced by HADOOP_LOGFILE. Using value of YARN_LOGFILE.
      WARNING: YARN_PID_DIR has been replaced by HADOOP_PID_DIR. Using value of YARN_PID_DIR.
      WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.
      17/11/21 03:21:58 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
      17/11/21 03:21:59 INFO client.RMProxy: Connecting to ResourceManager at ctr-e134-1499953498516-324910-01-000003.example.com/172.27.47.1:8050
      17/11/21 03:22:00 WARN shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
      17/11/21 03:22:00 INFO client.RMProxy: Connecting to ResourceManager at ctr-e134-1499953498516-324910-01-000003.example.com/172.27.47.1:8050
      17/11/21 03:22:00 INFO client.ServiceClient: Loading service definition from local FS: /usr/hdp/3.0.0.0-493/hadoop-yarn/yarn-service-examples/sleeper/sleeper.json
      17/11/21 03:22:01 INFO client.ServiceClient: Persisted service gour-sleeper at hdfs://ctr-e134-1499953498516-324910-01-000003.example.com:8020/user/hdfs/.yarn/services/gour-sleeper/gour-sleeper.json
      17/11/21 03:22:01 INFO conf.Configuration: resource-types.xml not found
      17/11/21 03:22:01 WARN client.ServiceClient: AM log4j property file doesn't exist: /usr/hdp/3.0.0.0-493/hadoop/conf/yarnservice-log4j.properties
      17/11/21 03:22:01 INFO client.ServiceClient: Uploading all dependency jars to HDFS. For faster submission of apps, pre-upload dependency jars to HDFS using command: yarn app -enableFastLaunch
      Exception in thread "main" java.io.FileNotFoundException: File /usr/hdp/3.0.0.0-493/hadoop/lib/ojdbc6.jar does not exist
      	at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:641)
      	at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:867)
      	at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:631)
      	at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:454)
      	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:365)
      	at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:2399)
      	at org.apache.hadoop.yarn.service.utils.CoreFileSystem.submitFile(CoreFileSystem.java:434)
      	at org.apache.hadoop.yarn.service.utils.ServiceUtils.putAllJars(ServiceUtils.java:409)
      	at org.apache.hadoop.yarn.service.provider.ProviderUtils.addAllDependencyJars(ProviderUtils.java:138)
      	at org.apache.hadoop.yarn.service.client.ServiceClient.addJarResource(ServiceClient.java:695)
      	at org.apache.hadoop.yarn.service.client.ServiceClient.submitApp(ServiceClient.java:553)
      	at org.apache.hadoop.yarn.service.client.ServiceClient.actionCreate(ServiceClient.java:212)
      	at org.apache.hadoop.yarn.service.client.ServiceClient.actionLaunch(ServiceClient.java:197)
      	at org.apache.hadoop.yarn.client.cli.ApplicationCLI.run(ApplicationCLI.java:447)
      	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
      	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
      	at org.apache.hadoop.yarn.client.cli.ApplicationCLI.main(ApplicationCLI.java:111)
      

        Attachments

        1. YARN-7543.01.patch
          3 kB
          Jian He

          Activity

            People

            • Assignee:
              jianhe Jian He
              Reporter:
              gsaha Gour Saha
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: