Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Environment:

      Windows + Cygwin

      Description

      When I try to run a datanode on Windows, I get the following exception:

      java.io.IOException: Expecting a line not the end of stream
      	at org.apache.hadoop.fs.DU.parseExecResult(DU.java:181)
      	at org.apache.hadoop.util.Shell.runCommand(Shell.java:179)
      	at org.apache.hadoop.util.Shell.run(Shell.java:134)
      	at org.apache.hadoop.fs.DU.<init>(DU.java:53)
      	at org.apache.hadoop.fs.DU.<init>(DU.java:63)
      	at org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.<init>(FSDataset.java:325)
      	at org.apache.hadoop.hdfs.server.datanode.FSDataset.<init>(FSDataset.java:681)
      	at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:291)
      	at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:205)
      	at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1238)
      	at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1193)
      

      This is because Hadoop execs "du -sk C:\tmp\hadoop-SYSTEM\dfs\data" with a Windows path representation, which cygwin du doesn't understand.

      C:\hudson>du -sk C:\tmp\hadoop-SYSTEM\dfs\data
      du -sk C:\tmp\hadoop-SYSTEM\dfs\data
      du: cannot access `C:\\tmp\\hadoop-SYSTEM\\dfs\\data': No such file or directory
      

      For this to work correctly, Hadoop would have to run cygpath first to get a Unix path representation, then to call DU.

      Also, I had to use the debugger to get this information. Shell.runCommand should catch IOException from parseExecResult and add the buffered stderr to simplify the error diagnostics.

      1. HADOOP-5486
        0.6 kB
        Kohsuke Kawaguchi

        Issue Links

          Activity

          Kohsuke Kawaguchi created issue -
          Doug Cutting made changes -
          Field Original Value New Value
          Assignee Kohsuke Kawaguchi [ kohsuke ]
          Kohsuke Kawaguchi made changes -
          Attachment HADOOP-5486 [ 12402304 ]
          Steve Loughran made changes -
          Link This issue is related to HADOOP-5875 [ HADOOP-5875 ]
          Owen O'Malley made changes -
          Project Hadoop Common [ 12310240 ] HDFS [ 12310942 ]
          Key HADOOP-5496 HDFS-197
          Affects Version/s 0.19.1 [ 12313473 ]
          Component/s dfs [ 12310710 ]
          Harsh J made changes -
          Assignee Kohsuke Kawaguchi [ kohsuke ] Harsh J [ qwertymaniac ]
          Harsh J made changes -
          Assignee Harsh J [ qwertymaniac ]
          Allen Wittenauer made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]

            People

            • Assignee:
              Unassigned
              Reporter:
              Kohsuke Kawaguchi
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development