Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-9865

FileContext.globStatus() has a regression with respect to relative path

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 2.3.0, 3.0.0-alpha1
    • 2.3.0
    • None
    • None

    Description

      I discovered the problem when running unit test TestMRJobClient on Windows. The cause is indirect in this case. In the unit test, we try to launch a job and list its status. The job failed, and caused the list command get a result of 0, which triggered the unit test assert. From the log and debug, the job failed because we failed to create the Jar with classpath (see code around FileUtil.createJarWithClassPath) in ContainerLaunch. This is a Windows specific step right now; so the test still passes on Linux. This step failed because we passed in a relative path to FileContext.globStatus() in FileUtil.createJarWithClassPath. The relevant log looks like the following.

      2013-08-12 16:12:05,937 WARN  [ContainersLauncher #0] launcher.ContainerLaunch (ContainerLaunch.java:call(270)) - Failed to launch container.
      org.apache.hadoop.HadoopIllegalArgumentException: Path is relative
      	at org.apache.hadoop.fs.Path.checkNotRelative(Path.java:74)
      	at org.apache.hadoop.fs.FileContext.getFSofPath(FileContext.java:304)
      	at org.apache.hadoop.fs.Globber.schemeFromPath(Globber.java:107)
      	at org.apache.hadoop.fs.Globber.glob(Globber.java:128)
      	at org.apache.hadoop.fs.FileContext$Util.globStatus(FileContext.java:1908)
      	at org.apache.hadoop.fs.FileUtil.createJarWithClassPath(FileUtil.java:1247)
      	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.sanitizeEnv(ContainerLaunch.java:679)
      	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:232)
      	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:1)
      	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      	at java.lang.Thread.run(Thread.java:662)
      

      I think this is a regression from HADOOP-9817. I modified some code and the unit test passed. (See the attached patch.) However, I think the impact is larger. I will add some unit tests to verify the behavior, and work on a more complete fix.

      Attachments

        1. HADOOP-9865-trunk.3.patch
          5 kB
          Chuan Liu
        2. HADOOP-9865-trunk.2.patch
          5 kB
          Chuan Liu
        3. HADOOP-9865-trunk.patch
          5 kB
          Chuan Liu
        4. HADOOP-9865-demo.patch
          0.9 kB
          Chuan Liu

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            chuanliu Chuan Liu
            chuanliu Chuan Liu
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Issue deployment