Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-9865

FileContext.globStatus() has a regression with respect to relative path

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.3.0, 3.0.0-alpha1
    • Fix Version/s: 2.3.0
    • Component/s: None
    • Labels:
      None
    • Target Version/s:

      Description

      I discovered the problem when running unit test TestMRJobClient on Windows. The cause is indirect in this case. In the unit test, we try to launch a job and list its status. The job failed, and caused the list command get a result of 0, which triggered the unit test assert. From the log and debug, the job failed because we failed to create the Jar with classpath (see code around FileUtil.createJarWithClassPath) in ContainerLaunch. This is a Windows specific step right now; so the test still passes on Linux. This step failed because we passed in a relative path to FileContext.globStatus() in FileUtil.createJarWithClassPath. The relevant log looks like the following.

      2013-08-12 16:12:05,937 WARN  [ContainersLauncher #0] launcher.ContainerLaunch (ContainerLaunch.java:call(270)) - Failed to launch container.
      org.apache.hadoop.HadoopIllegalArgumentException: Path is relative
      	at org.apache.hadoop.fs.Path.checkNotRelative(Path.java:74)
      	at org.apache.hadoop.fs.FileContext.getFSofPath(FileContext.java:304)
      	at org.apache.hadoop.fs.Globber.schemeFromPath(Globber.java:107)
      	at org.apache.hadoop.fs.Globber.glob(Globber.java:128)
      	at org.apache.hadoop.fs.FileContext$Util.globStatus(FileContext.java:1908)
      	at org.apache.hadoop.fs.FileUtil.createJarWithClassPath(FileUtil.java:1247)
      	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.sanitizeEnv(ContainerLaunch.java:679)
      	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:232)
      	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:1)
      	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      	at java.lang.Thread.run(Thread.java:662)
      

      I think this is a regression from HADOOP-9817. I modified some code and the unit test passed. (See the attached patch.) However, I think the impact is larger. I will add some unit tests to verify the behavior, and work on a more complete fix.

        Attachments

        1. HADOOP-9865-trunk.patch
          5 kB
          Chuan Liu
        2. HADOOP-9865-trunk.3.patch
          5 kB
          Chuan Liu
        3. HADOOP-9865-trunk.2.patch
          5 kB
          Chuan Liu
        4. HADOOP-9865-demo.patch
          0.9 kB
          Chuan Liu

          Issue Links

            Activity

              People

              • Assignee:
                chuanliu Chuan Liu
                Reporter:
                chuanliu Chuan Liu
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: