Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-2931

PublicLocalizer may fail until directory is initialized by LocalizeRunner

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.7.0
    • Component/s: nodemanager
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      When the data directory is cleaned up and NM is started with existing recovery state, because of YARN-90, it will not recreate the local dirs.
      This causes a PublicLocalizer to fail until getInitializedLocalDirs is called due to some LocalizeRunner for private localization.

      Example error

      2014-12-02 22:57:32,629 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Failed to download rsrc { { hdfs:/<blah machine>:8020/tmp/hive-hive/hive_2014-12-02_22-56-58_741_2045919883676051996-3/-mr-10004/8060c9dd-54b6-42fc-9d77-34b655fa5e82/reduce.xml, 1417589819618, FILE, null },pending,[(container_1417589109512_0001_02_000003)],119413444132127,DOWNLOADING}
      java.io.FileNotFoundException: File /data/yarn/nm/filecache does not exist
      	at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:524)
      	at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:737)
      	at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:514)
      	at org.apache.hadoop.fs.FileSystem.primitiveMkdir(FileSystem.java:1051)
      	at org.apache.hadoop.fs.DelegateToFileSystem.mkdir(DelegateToFileSystem.java:162)
      	at org.apache.hadoop.fs.FilterFs.mkdir(FilterFs.java:197)
      	at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:724)
      	at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:720)
      	at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
      	at org.apache.hadoop.fs.FileContext.mkdir(FileContext.java:720)
      	at org.apache.hadoop.yarn.util.FSDownload.createDir(FSDownload.java:104)
      	at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:351)
      	at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:60)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:745)
      2014-12-02 22:57:32,629 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1417589109512_0001_02_000003 transitioned from LOCALIZING to LOCALIZATION_FAILED
      

        Attachments

        1. YARN-2931.001.patch
          6 kB
          Anubhav Dhoot
        2. YARN-2931.002.patch
          6 kB
          Anubhav Dhoot
        3. YARN-2931.002.patch
          6 kB
          Anubhav Dhoot
        4. YARN-2931.003.patch
          8 kB
          Anubhav Dhoot
        5. YARN-2931.004.patch
          8 kB
          Anubhav Dhoot
        6. YARN-2931.005.patch
          8 kB
          Anubhav Dhoot

          Activity

            People

            • Assignee:
              adhoot Anubhav Dhoot
              Reporter:
              adhoot Anubhav Dhoot
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: