Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.14.0
    • Fix Version/s: 0.15.0
    • Component/s: fs
    • Labels:
      None

      Description

      My code use to work with previous version of hadoop, I upgraded to 0.14 and now:
      java.io.IOException: CreateProcess: df -k "C:\Documents and Settings\Benjamin\Local Settings\Temp\test14906test\mapredLocal" error=2
      at java.lang.ProcessImpl.create(Native Method)
      at java.lang.ProcessImpl.<init>(Unknown Source)
      at java.lang.ProcessImpl.start(Unknown Source)
      at java.lang.ProcessBuilder.start(Unknown Source)
      at java.lang.Runtime.exec(Unknown Source)
      at java.lang.Runtime.exec(Unknown Source)
      at org.apache.hadoop.fs.DF.doDF(DF.java:60)
      at org.apache.hadoop.fs.DF.<init>(DF.java:53)
      at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:198)
      at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:235)
      at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
      at org.apache.hadoop.mapred.MapOutputFile.getSpillFileForWrite(MapOutputFile.java:88)
      at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpillToDisk(MapTask.java:373)
      at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:593)
      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:190)
      at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:137)
      at org.apache.hadoop.mapred.LocalJobRunner.submitJob(LocalJobRunner.java:283)
      at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:397)
      ...

        Activity

        Benjamin Francisoud created issue -
        Benjamin Francisoud made changes -
        Field Original Value New Value
        Component/s fs [ 12310689 ]
        Fix Version/s 0.14.0 [ 12312474 ]
        Affects Version/s 0.14.0 [ 12312474 ]
        Hide
        Doug Cutting added a comment -

        Is anyone else seeing this? There were changes to DF.java in 0.14, but none that should cause this. Are you still running under a Cygwin shell? We depend on Cygwin binaries being on $PATH, including at least 'df' and 'bash'.

        Show
        Doug Cutting added a comment - Is anyone else seeing this? There were changes to DF.java in 0.14, but none that should cause this. Are you still running under a Cygwin shell? We depend on Cygwin binaries being on $PATH, including at least 'df' and 'bash'.
        Hide
        Benjamin Francisoud added a comment -

        It use to work without cygwin in the path var.

        I added it and it work yes.

        But I don't understand why it use to work before, that's weird ?!

        Show
        Benjamin Francisoud added a comment - It use to work without cygwin in the path var. I added it and it work yes. But I don't understand why it use to work before, that's weird ?!
        Hide
        Olivier Dagenais added a comment -

        I got this problem too: It appears DF is being used to compute available disk space before creating files. While Java does not appears to have support for this (see bug 4057701), it does appear that another Apache project (sort of) does:

        org.apache.commons.FileSystemUtils.freeSpaceKb

        ...at least it wouldn't require that Hadoop users on Windows install Cygwin.

        Show
        Olivier Dagenais added a comment - I got this problem too: It appears DF is being used to compute available disk space before creating files. While Java does not appears to have support for this (see bug 4057701 ), it does appear that another Apache project (sort of) does: org.apache.commons.FileSystemUtils.freeSpaceKb ...at least it wouldn't require that Hadoop users on Windows install Cygwin.
        Robert Chansler made changes -
        Assignee Mahadev konar [ mahadev ]
        Hide
        Mahadev konar added a comment -

        looks like this has been fixed and works with cygwin as intended. resolving for now.

        Show
        Mahadev konar added a comment - looks like this has been fixed and works with cygwin as intended. resolving for now.
        Mahadev konar made changes -
        Resolution Fixed [ 1 ]
        Status Open [ 1 ] Resolved [ 5 ]
        Fix Version/s 0.15.0 [ 12312565 ]
        Hide
        Prem Raj added a comment -

        I had this problem and the comments here were not very clear. I solved it by trying various things...so here is how I got it to work:

        My environment:

        • Windows XP
        • Eclipse running under Windows
        • Eclipse includes the IBM plugin for mapreduce/hadoop
        • I was trying to run the mapreduce job as a java application via Eclipse

        Solution:

        • Installed the latest stable version of Cygwin
        • Added "c:\Cygwin\bin " (or equivalent) to Path on windows
        • Started the Cygwin shell
        • Ran the mapreduce job as a java application while the Cygwin shell was running.

        Hope this helps clarify things for other newbies.

        Show
        Prem Raj added a comment - I had this problem and the comments here were not very clear. I solved it by trying various things...so here is how I got it to work: My environment: Windows XP Eclipse running under Windows Eclipse includes the IBM plugin for mapreduce/hadoop I was trying to run the mapreduce job as a java application via Eclipse Solution: Installed the latest stable version of Cygwin Added "c:\Cygwin\bin " (or equivalent) to Path on windows Started the Cygwin shell Ran the mapreduce job as a java application while the Cygwin shell was running. Hope this helps clarify things for other newbies.
        Hide
        steve added a comment -

        Hello, I've done all this, but when I run my job in eclipse, I get the Visual Studio just-in-time debugger for df.exe, as if it is finding it, but it's not running correctly.
        Has anyone else had this problem?
        It seems to occur when the map job completes...

        -Steve

        Show
        steve added a comment - Hello, I've done all this, but when I run my job in eclipse, I get the Visual Studio just-in-time debugger for df.exe, as if it is finding it, but it's not running correctly. Has anyone else had this problem? It seems to occur when the map job completes... -Steve

          People

          • Assignee:
            Mahadev konar
            Reporter:
            Benjamin Francisoud
          • Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development