Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Environment:

      Windows + Cygwin

      Description

      When I try to run a datanode on Windows, I get the following exception:

      java.io.IOException: Expecting a line not the end of stream
      	at org.apache.hadoop.fs.DU.parseExecResult(DU.java:181)
      	at org.apache.hadoop.util.Shell.runCommand(Shell.java:179)
      	at org.apache.hadoop.util.Shell.run(Shell.java:134)
      	at org.apache.hadoop.fs.DU.<init>(DU.java:53)
      	at org.apache.hadoop.fs.DU.<init>(DU.java:63)
      	at org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.<init>(FSDataset.java:325)
      	at org.apache.hadoop.hdfs.server.datanode.FSDataset.<init>(FSDataset.java:681)
      	at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:291)
      	at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:205)
      	at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1238)
      	at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1193)
      

      This is because Hadoop execs "du -sk C:\tmp\hadoop-SYSTEM\dfs\data" with a Windows path representation, which cygwin du doesn't understand.

      C:\hudson>du -sk C:\tmp\hadoop-SYSTEM\dfs\data
      du -sk C:\tmp\hadoop-SYSTEM\dfs\data
      du: cannot access `C:\\tmp\\hadoop-SYSTEM\\dfs\\data': No such file or directory
      

      For this to work correctly, Hadoop would have to run cygpath first to get a Unix path representation, then to call DU.

      Also, I had to use the debugger to get this information. Shell.runCommand should catch IOException from parseExecResult and add the buffered stderr to simplify the error diagnostics.

      1. HADOOP-5486
        0.6 kB
        Kohsuke Kawaguchi

        Issue Links

          Activity

          No work has yet been logged on this issue.

            People

            • Assignee:
              Unassigned
              Reporter:
              Kohsuke Kawaguchi
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development