Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-9865

FileContext.globStatus() has a regression with respect to relative path

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 2.3.0, 3.0.0-alpha1
    • 2.3.0
    • None
    • None

    Description

      I discovered the problem when running unit test TestMRJobClient on Windows. The cause is indirect in this case. In the unit test, we try to launch a job and list its status. The job failed, and caused the list command get a result of 0, which triggered the unit test assert. From the log and debug, the job failed because we failed to create the Jar with classpath (see code around FileUtil.createJarWithClassPath) in ContainerLaunch. This is a Windows specific step right now; so the test still passes on Linux. This step failed because we passed in a relative path to FileContext.globStatus() in FileUtil.createJarWithClassPath. The relevant log looks like the following.

      2013-08-12 16:12:05,937 WARN  [ContainersLauncher #0] launcher.ContainerLaunch (ContainerLaunch.java:call(270)) - Failed to launch container.
      org.apache.hadoop.HadoopIllegalArgumentException: Path is relative
      	at org.apache.hadoop.fs.Path.checkNotRelative(Path.java:74)
      	at org.apache.hadoop.fs.FileContext.getFSofPath(FileContext.java:304)
      	at org.apache.hadoop.fs.Globber.schemeFromPath(Globber.java:107)
      	at org.apache.hadoop.fs.Globber.glob(Globber.java:128)
      	at org.apache.hadoop.fs.FileContext$Util.globStatus(FileContext.java:1908)
      	at org.apache.hadoop.fs.FileUtil.createJarWithClassPath(FileUtil.java:1247)
      	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.sanitizeEnv(ContainerLaunch.java:679)
      	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:232)
      	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:1)
      	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      	at java.lang.Thread.run(Thread.java:662)
      

      I think this is a regression from HADOOP-9817. I modified some code and the unit test passed. (See the attached patch.) However, I think the impact is larger. I will add some unit tests to verify the behavior, and work on a more complete fix.

      Attachments

        1. HADOOP-9865-demo.patch
          0.9 kB
          Chuan Liu
        2. HADOOP-9865-trunk.2.patch
          5 kB
          Chuan Liu
        3. HADOOP-9865-trunk.3.patch
          5 kB
          Chuan Liu
        4. HADOOP-9865-trunk.patch
          5 kB
          Chuan Liu

        Issue Links

          Activity

            People

              chuanliu Chuan Liu
              chuanliu Chuan Liu
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: