df command doesn't exist under windows

• Key: HADOOP-1792
• Type: Bug
• Status: Resolved
• Resolution: Fixed
• Priority: Major
• Assignee: Mahadev konar
• Reporter: Benjamin Francisoud
• Component/s: fs
• Affects Version/s: 0.14.0
• Fix Version/s: 0.15.0

Description

My code use to work with previous version of hadoop, I upgraded to 0.14 and now:
java.io.IOException: CreateProcess: df -k "C:\Documents and Settings\Benjamin\Local Settings\Temp\test14906test\mapredLocal" error=2
at java.lang.ProcessImpl.create(Native Method)
at java.lang.ProcessImpl.<init>(Unknown Source)
at java.lang.ProcessImpl.start(Unknown Source)
at java.lang.ProcessBuilder.start(Unknown Source)
at java.lang.Runtime.exec(Unknown Source)
at java.lang.Runtime.exec(Unknown Source)
at org.apache.hadoop.fs.DF.doDF(DF.java:60)
at org.apache.hadoop.fs.DF.<init>(DF.java:53)
at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:198)
at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:235)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
at org.apache.hadoop.mapred.MapOutputFile.getSpillFileForWrite(MapOutputFile.java:88)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpillToDisk(MapTask.java:373)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:593)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:190)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:137)
at org.apache.hadoop.mapred.LocalJobRunner.submitJob(LocalJobRunner.java:283)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:397)
...

Comments

[Doug Cutting]

Is anyone else seeing this? There were changes to DF.java in 0.14, but none that should cause this. Are you still running under a Cygwin shell? We depend on Cygwin binaries being on $PATH, including at least 'df' and 'bash'.

[Benjamin Francisoud]

It use to work without cygwin in the path var.

I added it and it work yes.

But I don't understand why it use to work before, that's weird ?!

[Olivier Dagenais]

I got this problem too: It appears DF is being used to compute available disk space before creating files. While Java does not appears to have support for this (see bug 4057701), it does appear that another Apache project (sort of) does:

org.apache.commons.FileSystemUtils.freeSpaceKb

...at least it wouldn't require that Hadoop users on Windows install Cygwin.

[Mahadev konar]

looks like this has been fixed and works with cygwin as intended. resolving for now.

[Prem Raj]

I had this problem and the comments here were not very clear. I solved it by trying various things...so here is how I got it to work:

My environment:

• Windows XP
• Eclipse running under Windows
• Eclipse includes the IBM plugin for mapreduce/hadoop
• I was trying to run the mapreduce job as a java application via Eclipse

Solution:

• Installed the latest stable version of Cygwin
• Added "c:\Cygwin\bin " (or equivalent) to Path on windows
• Started the Cygwin shell
• Ran the mapreduce job as a java application while the Cygwin shell was running.

Hope this helps clarify things for other newbies.

[steve]

Hello, I've done all this, but when I run my job in eclipse, I get the Visual Studio just-in-time debugger for df.exe, as if it is finding it, but it's not running correctly.
Has anyone else had this problem?
It seems to occur when the map job completes...

-Steve