Hadoop Common
  1. Hadoop Common
  2. HADOOP-7682

taskTracker could not start because "Failed to set permissions" to "ttprivate to 0700"

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 1.0.1
    • Fix Version/s: None
    • Component/s: fs
    • Labels:
      None
    • Environment:

      OS:WindowsXP SP3 , Filesystem :NTFS, cygwin 1.7.9-1, jdk1.6.0_05

    • Target Version/s:
    • Tags:
      ttprivate 0700 task tracker cygwin

      Description

      ERROR org.apache.hadoop.mapred.TaskTracker:Can not start task tracker because java.io.IOException:Failed to set permissions of path:/tmp/hadoop-cyg_server/mapred/local/ttprivate to 0700
      at org.apache.hadoop.fs.RawLocalFileSystem.checkReturnValue(RawLocalFileSystem.java:525)
      at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:499)
      at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:318)
      at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:183)
      at org.apache.hadoop.mapred.TaskTracker.initialize(TaskTracker.java:635)
      at org.apache.hadoop.mapred.TaskTracker.(TaskTracker.java:1328)
      at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3430)

      Since hadoop0.20.203 when the TaskTracker initialize, it checks the permission(TaskTracker Line 624) of (org.apache.hadoop.mapred.TaskTracker.TT_LOG_TMP_DIR,org.apache.hadoop.mapred.TaskTracker.TT_PRIVATE_DIR, org.apache.hadoop.mapred.TaskTracker.TT_PRIVATE_DIR).RawLocalFileSystem(http://svn.apache.org/viewvc/hadoop/common/tags/release-0.20.203.0/src/core/org/apache/hadoop/fs/RawLocalFileSystem.java?view=markup) call setPermission(Line 481) to deal with it, setPermission works fine on *nx, however,it dose not alway works on windows.

      setPermission call setReadable of Java.io.File in the line 498, but according to the Table1 below provided by oracle,setReadable(false) will always return false on windows, the same as setExecutable(false).

      http://java.sun.com/developer/technicalArticles/J2SE/Desktop/javase6/enhancements/

      is it cause the task tracker "Failed to set permissions" to "ttprivate to 0700"?
      Hadoop 0.20.202 works fine in the same environment.

        Issue Links

          Activity

          Hide
          Matthieu Lecoutre added a comment -

          Thanks Magic Xie,
          Same problem on 0.20.204 on windows

          Show
          Matthieu Lecoutre added a comment - Thanks Magic Xie, Same problem on 0.20.204 on windows
          Hide
          Todd Fast added a comment -

          Also a problem on 0.20.205. Problem exists with or without running Hadoop with Cygwin. The only fix seems to be to downgrade to 0.20.2.

          Additional comments around the Web on this bug:
          http://comments.gmane.org/gmane.comp.jakarta.lucene.hadoop.user/25837
          http://lucene.472066.n3.nabble.com/SimpleKMeansCLustering-quot-Failed-to-set-permissions-of-path-to-0700-quot-td3429867.html

          Show
          Todd Fast added a comment - Also a problem on 0.20.205. Problem exists with or without running Hadoop with Cygwin. The only fix seems to be to downgrade to 0.20.2. Additional comments around the Web on this bug: http://comments.gmane.org/gmane.comp.jakarta.lucene.hadoop.user/25837 http://lucene.472066.n3.nabble.com/SimpleKMeansCLustering-quot-Failed-to-set-permissions-of-path-to-0700-quot-td3429867.html
          Hide
          Vijaya Phanindra added a comment -

          The problem seems to exists for release 1.0.0 also. Works fine if downgraded to 0.20.2

          java.io.IOException: Failed to set permissions of path: \tmp\hadoop-UserName\mapred\staging\UserName-1687815415\.staging to 0700
          at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:682)
          at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:655)
          at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
          at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
          at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189)
          at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:116)
          at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:856)
          at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:850)
          at java.security.AccessController.doPrivileged(Native Method)
          at javax.security.auth.Subject.doAs(Subject.java:396)
          at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083)
          at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:850)
          at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:824)
          at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1261)
          at org.apache.hadoop.examples.Grep.run(Grep.java:69)
          at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
          at org.apache.hadoop.examples.Grep.main(Grep.java:93)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
          at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
          at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

          Show
          Vijaya Phanindra added a comment - The problem seems to exists for release 1.0.0 also. Works fine if downgraded to 0.20.2 java.io.IOException: Failed to set permissions of path: \tmp\hadoop-UserName\mapred\staging\UserName-1687815415\.staging to 0700 at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:682) at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:655) at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509) at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344) at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189) at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:116) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:856) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:850) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083) at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:850) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:824) at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1261) at org.apache.hadoop.examples.Grep.run(Grep.java:69) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.examples.Grep.main(Grep.java:93) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139) at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
          Hide
          David Eagen added a comment -

          This worked on 0.22.0.

          Show
          David Eagen added a comment - This worked on 0.22.0.
          Hide
          tigar added a comment -

          same problem with cygin~'

          Show
          tigar added a comment - same problem with cygin~ '
          Hide
          tigar added a comment -

          ps: my hadoop version is 1.0.1~

          Show
          tigar added a comment - ps: my hadoop version is 1.0.1~
          Hide
          FKorning added a comment -

          There's a bunch of issues at work. I've patched this up locally
          on my own 1.0.2-SNAPSHOT, but it takes a lot of yak-shaving to fix.

          First you need to set up hadoop-1.0.1 including source, ant, ivy,
          and cygwin with ssh/ssl and tcp_wrappers.

          Then use sshd_config to create a cyg_server priviledged user.
          From an admin cygwin shell, you then have to edit the /etc/passwd
          file and give that user a valid shell and user home, change the
          password for the user, and finally generate ssh keys for the user
          and copy the user's id_rsa.pub public key into ~/.ssh/authorized_keys.

          if done right you should be able to ssh cyg_server@localhost.

          Now the main problem is a confusion between the hadoop shell scripts
          that expect unix paths like /tmp, and the haddop java binaries who
          interpret this path as C:\tmp.

          Unfortunately, neither Cygwin symlinks nor even Windows NT Junctions
          are supported by the java io filesystem. Thus the only way to get
          around this is to enforce the cygwin paths to be identical to windows
          paths.

          I get around this by creating a circular symlink in "/cygwin" -> "/".
          To avoid confusion with "C:" drive mappings, all my paths are relative.
          This means that windows "\cygwin\tmp" equals cygwin's "/cygwin/tmp".

          For pid files use /cygwin/tmp/
          For tmp file use /cygwin/tmp/haddop-$

          {USER}/
          For log files use /cygwin/tmp/haddop-${USER}

          /logs/

          First the ssh slaves invocation warpper is broken because it fails to
          provide the user's ssh login, which isn't defaulted to in cygwin openssh.

          slaves.sh:

          for slave in `cat "$HOSTLIST"|sed "s/#.*$//;/^$/d"`; do
          ssh -l $USER $HADOOP_SSH_OPTS $slave $"$

          {@// /\\ }

          " \
          2>&1 | sed "s/^/$slave: /" &
          if [ "$HADOOP_SLAVE_SLEEP" != "" ]; then
          sleep $HADOOP_SLAVE_SLEEP
          fi
          done

          Next the hadoop shell scripts are broken. you need to fix the environments
          for cygwin paths in hadoop-env.sh, and then make sure this file is invoked
          by both hadoop-config.sh, and finally the hadoop* sh wrapper script. For me
          its JRE java invocation was also broken, so I provide the whole srcript below.

          hadoop-env.sh:

          HADOOP_PID_DIR=/cygwin/tmp/
          HADOOP_TMP_DIR=/cygwin/tmp/hadoop-$

          {USER}
          HADOOP_LOG_DIR=/cygwin/tmp/hadoop-${USER}

          /logs

          hadoop (sh):

          #!/usr/bin/env bash

          1. Licensed to the Apache Software Foundation (ASF) under one or more
          2. contributor license agreements. See the NOTICE file distributed with
          3. this work for additional information regarding copyright ownership.
          4. The ASF licenses this file to You under the Apache License, Version 2.0
          5. (the "License"); you may not use this file except in compliance with
          6. the License. You may obtain a copy of the License at
            #
          7. http://www.apache.org/licenses/LICENSE-2.0
            #
          8. Unless required by applicable law or agreed to in writing, software
          9. distributed under the License is distributed on an "AS IS" BASIS,
          10. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          11. See the License for the specific language governing permissions and
          12. limitations under the License.
          1. The Hadoop command script
            #
          2. Environment Variables
            #
          3. JAVA_HOME The java implementation to use. Overrides JAVA_HOME.
            #
          4. HADOOP_CLASSPATH Extra Java CLASSPATH entries.
            #
          5. HADOOP_USER_CLASSPATH_FIRST When defined, the HADOOP_CLASSPATH is
          6. added in the beginning of the global
          7. classpath. Can be defined, for example,
          8. by doing
          9. export HADOOP_USER_CLASSPATH_FIRST=true
            #
          10. HADOOP_HEAPSIZE The maximum amount of heap to use, in MB.
          11. Default is 1000.
            #
          12. HADOOP_OPTS Extra Java runtime options.
          13. HADOOP_NAMENODE_OPTS These options are added to HADOOP_OPTS
          14. HADOOP_CLIENT_OPTS when the respective command is run.
          15. HADOOP_ {COMMAND}

            _OPTS etc HADOOP_JT_OPTS applies to JobTracker

          16. for e.g. HADOOP_CLIENT_OPTS applies to
          17. more than one command (fs, dfs, fsck,
          18. dfsadmin etc)
            #
          19. HADOOP_CONF_DIR Alternate conf dir. Default is $ {HADOOP_HOME}/conf.
            #
            # HADOOP_ROOT_LOGGER The root appender. Default is INFO,console
            #

            bin=`dirname "$0"`
            bin=`cd "$bin"; pwd`

            cygwin=false
            case "`uname`" in
            CYGWIN*) cygwin=true;;
            esac


            if [ -e "$bin"/../libexec/hadoop-config.sh ]; then
            . "$bin"/../libexec/hadoop-config.sh
            else
            . "$bin"/hadoop-config.sh
            fi


            # if no args specified, show usage
            if [ $# = 0 ]; then
            echo "Usage: hadoop [--config confdir] COMMAND"
            echo "where COMMAND is one of:"
            echo " namenode -format format the DFS filesystem"
            echo " secondarynamenode run the DFS secondary namenode"
            echo " namenode run the DFS namenode"
            echo " datanode run a DFS datanode"
            echo " dfsadmin run a DFS admin client"
            echo " mradmin run a Map-Reduce admin client"
            echo " fsck run a DFS filesystem checking utility"
            echo " fs run a generic filesystem user client"
            echo " balancer run a cluster balancing utility"
            echo " fetchdt fetch a delegation token from the NameNode"
            echo " jobtracker run the MapReduce job Tracker node"
            echo " pipes run a Pipes job"
            echo " tasktracker run a MapReduce task Tracker node"
            echo " historyserver run job history servers as a standalone daemon"
            echo " job manipulate MapReduce jobs"
            echo " queue get information regarding JobQueues"
            echo " version print the version"
            echo " jar <jar> run a jar file"
            echo " distcp <srcurl> <desturl> copy file or directories recursively"
            echo " archive -archiveName NAME -p <parent path> <src>* <dest> create a hadoop archive"
            echo " classpath prints the class path needed to get the"
            echo " Hadoop jar and the required libraries"
            echo " daemonlog get/set the log level for each daemon"
            echo " or"
            echo " CLASSNAME run the class named CLASSNAME"
            echo "Most commands print help when invoked w/o parameters."
            exit 1
            fi

            # get arguments
            COMMAND=$1
            shift

            # Determine if we're starting a secure datanode, and if so, redefine appropriate variables
            if [ "$COMMAND" == "datanode" ] && [ "$EUID" -eq 0 ] && [ -n "$HADOOP_SECURE_DN_USER" ]; then
            HADOOP_PID_DIR=$HADOOP_SECURE_DN_PID_DIR
            HADOOP_LOG_DIR=$HADOOP_SECURE_DN_LOG_DIR
            HADOOP_IDENT_STRING=$HADOOP_SECURE_DN_USER
            starting_secure_dn="true"
            fi

            if [ "$JAVA_HOME" != "" ]; then
            #echo "JAVA_HOME: $JAVA_HOME"
            JAVA_HOME="$JAVA_HOME"
            fi
            # some Java parameters
            if $cygwin; then
            JAVA_HOME=`cygpath -w "$JAVA_HOME"`
            #echo "cygwin JAVA_HOME: $JAVA_HOME"
            fi
            if [ "$JAVA_HOME" == "" ]; then
            echo "Error: JAVA_HOME is not set: $JAVA_HOME"
            exit 1
            fi

            JAVA=$JAVA_HOME/bin/java
            JAVA_HEAP_MAX=-Xmx1000m

            # check envvars which might override default args
            if [ "$HADOOP_HEAPSIZE" != "" ]; then
            #echo "run with heapsize $HADOOP_HEAPSIZE"
            JAVA_HEAP_MAX="-Xmx""$HADOOP_HEAPSIZE""m"
            #echo $JAVA_HEAP_MAX
            fi

            # CLASSPATH initially contains $HADOOP_CONF_DIR
            CLASSPATH="${HADOOP_CONF_DIR}"
            if [ "$HADOOP_USER_CLASSPATH_FIRST" != "" ] && [ "$HADOOP_CLASSPATH" != "" ] ; then
            CLASSPATH=${CLASSPATH}:${HADOOP_CLASSPATH}
            fi
            CLASSPATH=${CLASSPATH}:$JAVA_HOME/lib/tools.jar

            # for developers, add Hadoop classes to CLASSPATH
            if [ -d "$HADOOP_HOME/build/classes" ]; then
            CLASSPATH=${CLASSPATH}:$HADOOP_HOME/build/classes
            fi
            if [ -d "$HADOOP_HOME/build/webapps" ]; then
            CLASSPATH=${CLASSPATH}:$HADOOP_HOME/build
            fi
            if [ -d "$HADOOP_HOME/build/test/classes" ]; then
            CLASSPATH=${CLASSPATH}:$HADOOP_HOME/build/test/classes
            fi
            if [ -d "$HADOOP_HOME/build/tools" ]; then
            CLASSPATH=${CLASSPATH}:$HADOOP_HOME/build/tools
            fi

            # so that filenames w/ spaces are handled correctly in loops below
            IFS=

            # for releases, add core hadoop jar & webapps to CLASSPATH
            if [ e $HADOOP_PREFIX/share/hadoop/hadoop-core* ]; then
            # binary layout
            if [ -d "$HADOOP_PREFIX/share/hadoop/webapps" ]; then
            CLASSPATH=${CLASSPATH}:$HADOOP_PREFIX/share/hadoop
            fi
            for f in $HADOOP_PREFIX/share/hadoop/hadoop-core-*.jar; do
            CLASSPATH=${CLASSPATH}:$f;
            done

            # add libs to CLASSPATH
            for f in $HADOOP_PREFIX/share/hadoop/lib/*.jar; do
            CLASSPATH=${CLASSPATH}:$f;
            done

            for f in $HADOOP_PREFIX/share/hadoop/lib/jsp-2.1/*.jar; do
            CLASSPATH=${CLASSPATH}:$f;
            done

            for f in $HADOOP_PREFIX/share/hadoop/hadoop-tools-*.jar; do
            TOOL_PATH=${TOOL_PATH}:$f;
            done
            else
            # tarball layout
            if [ -d "$HADOOP_HOME/webapps" ]; then
            CLASSPATH=${CLASSPATH}:$HADOOP_HOME
            fi
            for f in $HADOOP_HOME/hadoop-core-*.jar; do
            CLASSPATH=${CLASSPATH}:$f;
            done

            # add libs to CLASSPATH
            for f in $HADOOP_HOME/lib/*.jar; do
            CLASSPATH=${CLASSPATH}:$f;
            done

            if [ -d "$HADOOP_HOME/build/ivy/lib/Hadoop/common" ]; then
            for f in $HADOOP_HOME/build/ivy/lib/Hadoop/common/*.jar; do
            CLASSPATH=${CLASSPATH}:$f;
            done
            fi

            for f in $HADOOP_HOME/lib/jsp-2.1/*.jar; do
            CLASSPATH=${CLASSPATH}:$f;
            done

            for f in $HADOOP_HOME/hadoop-tools-*.jar; do
            TOOL_PATH=${TOOL_PATH}:$f;
            done
            for f in $HADOOP_HOME/build/hadoop-tools-*.jar; do
            TOOL_PATH=${TOOL_PATH}:$f;
            done
            fi

            # add user-specified CLASSPATH last
            if [ "$HADOOP_USER_CLASSPATH_FIRST" = "" ] && [ "$HADOOP_CLASSPATH" != "" ]; then
            CLASSPATH=${CLASSPATH}:${HADOOP_CLASSPATH}
            fi

            # default log directory & file
            if [ "$HADOOP_LOG_DIR" = "" ]; then
            HADOOP_LOG_DIR="$HADOOP_HOME/logs"
            fi
            if [ "$HADOOP_LOGFILE" = "" ]; then
            HADOOP_LOGFILE='hadoop.log'
            fi

            # default policy file for service-level authorization
            if [ "$HADOOP_POLICYFILE" = "" ]; then
            HADOOP_POLICYFILE="hadoop-policy.xml"
            fi

            # restore ordinary behaviour
            unset IFS

            # figure out which class to run
            if [ "$COMMAND" = "classpath" ] ; then
            if $cygwin; then
            CLASSPATH=`cygpath -wp "$CLASSPATH"`
            fi
            echo $CLASSPATH
            exit
            elif [ "$COMMAND" = "namenode" ] ; then
            CLASS='org.apache.hadoop.hdfs.server.namenode.NameNode'
            HADOOP_OPTS="$HADOOP_OPTS $HADOOP_NAMENODE_OPTS"
            elif [ "$COMMAND" = "secondarynamenode" ] ; then
            CLASS='org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode'
            HADOOP_OPTS="$HADOOP_OPTS $HADOOP_SECONDARYNAMENODE_OPTS"
            elif [ "$COMMAND" = "datanode" ] ; then
            CLASS='org.apache.hadoop.hdfs.server.datanode.DataNode'
            if [ "$starting_secure_dn" = "true" ]; then
            HADOOP_OPTS="$HADOOP_OPTS -jvm server $HADOOP_DATANODE_OPTS"
            else
            HADOOP_OPTS="$HADOOP_OPTS -server $HADOOP_DATANODE_OPTS"
            fi
            elif [ "$COMMAND" = "fs" ] ; then
            CLASS=org.apache.hadoop.fs.FsShell
            HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS"
            elif [ "$COMMAND" = "dfs" ] ; then
            CLASS=org.apache.hadoop.fs.FsShell
            HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS"
            elif [ "$COMMAND" = "dfsadmin" ] ; then
            CLASS=org.apache.hadoop.hdfs.tools.DFSAdmin
            HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS"
            elif [ "$COMMAND" = "mradmin" ] ; then
            CLASS=org.apache.hadoop.mapred.tools.MRAdmin
            HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS"
            elif [ "$COMMAND" = "fsck" ] ; then
            CLASS=org.apache.hadoop.hdfs.tools.DFSck
            HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS"
            elif [ "$COMMAND" = "balancer" ] ; then
            CLASS=org.apache.hadoop.hdfs.server.balancer.Balancer
            HADOOP_OPTS="$HADOOP_OPTS $HADOOP_BALANCER_OPTS"
            elif [ "$COMMAND" = "fetchdt" ] ; then
            CLASS=org.apache.hadoop.hdfs.tools.DelegationTokenFetcher
            elif [ "$COMMAND" = "jobtracker" ] ; then
            CLASS=org.apache.hadoop.mapred.JobTracker
            HADOOP_OPTS="$HADOOP_OPTS $HADOOP_JOBTRACKER_OPTS"
            elif [ "$COMMAND" = "historyserver" ] ; then
            CLASS=org.apache.hadoop.mapred.JobHistoryServer
            HADOOP_OPTS="$HADOOP_OPTS $HADOOP_JOB_HISTORYSERVER_OPTS"
            elif [ "$COMMAND" = "tasktracker" ] ; then
            CLASS=org.apache.hadoop.mapred.TaskTracker
            HADOOP_OPTS="$HADOOP_OPTS $HADOOP_TASKTRACKER_OPTS"
            elif [ "$COMMAND" = "job" ] ; then
            CLASS=org.apache.hadoop.mapred.JobClient
            HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS"
            elif [ "$COMMAND" = "queue" ] ; then
            CLASS=org.apache.hadoop.mapred.JobQueueClient
            HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS"
            elif [ "$COMMAND" = "pipes" ] ; then
            CLASS=org.apache.hadoop.mapred.pipes.Submitter
            HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS"
            elif [ "$COMMAND" = "version" ] ; then
            CLASS=org.apache.hadoop.util.VersionInfo
            HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS"
            elif [ "$COMMAND" = "jar" ] ; then
            CLASS=org.apache.hadoop.util.RunJar
            HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS"
            elif [ "$COMMAND" = "distcp" ] ; then
            CLASS=org.apache.hadoop.tools.DistCp
            CLASSPATH=${CLASSPATH}:${TOOL_PATH}
            HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS"
            elif [ "$COMMAND" = "daemonlog" ] ; then
            CLASS=org.apache.hadoop.log.LogLevel
            HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS"
            elif [ "$COMMAND" = "archive" ] ; then
            CLASS=org.apache.hadoop.tools.HadoopArchives
            CLASSPATH=${CLASSPATH}:${TOOL_PATH}
            HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS"
            elif [ "$COMMAND" = "sampler" ] ; then
            CLASS=org.apache.hadoop.mapred.lib.InputSampler
            HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS"
            else
            CLASS=$COMMAND
            fi


            # cygwin path translation
            if $cygwin; then
            JAVA_HOME=`cygpath -w "$JAVA_HOME"`
            CLASSPATH=`cygpath -wp "$CLASSPATH"`
            HADOOP_HOME=`cygpath -w "$HADOOP_HOME"`
            HADOOP_LOG_DIR=`cygpath -w "$HADOOP_LOG_DIR"`
            TOOL_PATH=`cygpath -wp "$TOOL_PATH"`
            fi

            # setup 'java.library.path' for native-hadoop code if necessary
            JAVA_LIBRARY_PATH=''


            if [ -d "${HADOOP_HOME}

            /build/native" -o -d "$

            {HADOOP_HOME}/lib/native" -o -e "${HADOOP_PREFIX}/lib/libhadoop.a" ]; then
            JAVA_PLATFORM=`${JAVA} -classpath ${CLASSPATH} -Xmx32m ${HADOOP_JAVA_PLATFORM_OPTS} org.apache.hadoop.util.PlatformName | sed -e "s/ /_/g"`
            #echo "JAVA_PLATFORM: $JAVA_PLATFORM"

            if [ "$JAVA_PLATFORM" = "Windows_7-amd64-64" ]; then
            JSVC_ARCH="amd64"
            elif [ "$JAVA_PLATFORM" = "Linux-amd64-64" ]; then
            JSVC_ARCH="amd64"
            else
            JSVC_ARCH="i386"
            fi

            if [ -d "$HADOOP_HOME/build/native" ]; then
            JAVA_LIBRARY_PATH=${HADOOP_HOME}

            /build/native/$

            {JAVA_PLATFORM}/lib
            fi

            if [ -d "${HADOOP_HOME}/lib/native" ]; then
            if [ "x$JAVA_LIBRARY_PATH" != "x" ]; then
            JAVA_LIBRARY_PATH=${JAVA_LIBRARY_PATH}:${HADOOP_HOME}/lib/native/${JAVA_PLATFORM}

            else
            JAVA_LIBRARY_PATH=$

            {HADOOP_HOME}

            /lib/native/$

            {JAVA_PLATFORM}

            fi
            fi

          if [ -e "$

          {HADOOP_PREFIX}/lib/libhadoop.a" ]; then
          JAVA_LIBRARY_PATH=${HADOOP_PREFIX}

          /lib
          fi
          fi

          1. cygwin path translation
            if $cygwin; then
            JAVA_LIBRARY_PATH=`cygpath -wp "$JAVA_LIBRARY_PATH"`
            PATH="/cygwin/bin:/cygwin/usr/bin:`cygpath -p $ {PATH}

            `"
            fi

          HADOOP_OPTS="$HADOOP_OPTS -Dhadoop.tmp.dir=$HADOOP_TMP_DIR"
          HADOOP_OPTS="$HADOOP_OPTS -Dhadoop.log.dir=$HADOOP_LOG_DIR"
          HADOOP_OPTS="$HADOOP_OPTS -Dhadoop.log.file=$HADOOP_LOGFILE"
          HADOOP_OPTS="$HADOOP_OPTS -Dhadoop.home.dir=$HADOOP_HOME"
          HADOOP_OPTS="$HADOOP_OPTS -Dhadoop.id.str=$HADOOP_IDENT_STRING"
          HADOOP_OPTS="$HADOOP_OPTS -Dhadoop.root.logger=$

          {HADOOP_ROOT_LOGGER:-INFO,console}

          "

          #turn security logger on the namenode and jobtracker only
          if [ $COMMAND = "namenode" ] || [ $COMMAND = "jobtracker" ]; then
          HADOOP_OPTS="$HADOOP_OPTS -Dhadoop.security.logger=$

          {HADOOP_SECURITY_LOGGER:-INFO,DRFAS}

          "
          else
          HADOOP_OPTS="$HADOOP_OPTS -Dhadoop.security.logger=$

          {HADOOP_SECURITY_LOGGER:-INFO,NullAppender}

          "
          fi

          if [ "x$JAVA_LIBRARY_PATH" != "x" ]; then
          HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=$JAVA_LIBRARY_PATH"
          fi
          HADOOP_OPTS="$HADOOP_OPTS -Dhadoop.policy.file=$HADOOP_POLICYFILE"

          1. Check to see if we should start a secure datanode
            if [ "$starting_secure_dn" = "true" ]; then
            if [ "$HADOOP_PID_DIR" = "" ]; then
            HADOOP_SECURE_DN_PID="/tmp/hadoop_secure_dn.pid"
            else
            HADOOP_SECURE_DN_PID="$HADOOP_PID_DIR/hadoop_secure_dn.pid"
            fi

          exec "$HADOOP_HOME/libexec/jsvc.$

          {JSVC_ARCH}

          " -Dproc_$COMMAND -outfile "$HADOOP_LOG_DIR/jsvc.out" \
          -errfile "$HADOOP_LOG_DIR/jsvc.err" \
          -pidfile "$HADOOP_SECURE_DN_PID" \
          -nodetach \
          -user "$HADOOP_SECURE_DN_USER" \
          -cp "$CLASSPATH" \
          $JAVA_HEAP_MAX $HADOOP_OPTS \
          org.apache.hadoop.hdfs.server.datanode.SecureDataNodeStarter "$@"
          else

          1. run it
            exec "$JAVA" -Dproc_$COMMAND $JAVA_HEAP_MAX $HADOOP_OPTS -classpath "$CLASSPATH" $CLASS "$@"
            fi

          Next the hadoop fs and utilities are broken, as they expect shells
          with POSIX /bin executables in their path (bash,chmod,chown,chgrp)
          For various reasons it's a real bad idea to add "/cygwin/bin" to your
          windows path, so we're going to have to fix the utility classes to
          be cygwin aware and use the "/cygwin/bin" binaries instead.

          This is why you need the source, because we're going to have to fix
          the java source and recompile the hadoop core libraries (and why you
          need ant ivy).


          Before we do this, the contrib Gridmix is broken as it uses a strange
          generic Enum code that just craps out in jdk/jre 1.7 and above.
          The fix is to dumb it down and use untyped Enums.

          Gridmix.java:

          /*
          private <T> String getEnumValues(Enum<? extends T>[] e) {
          StringBuilder sb = new StringBuilder();
          String sep = "";
          for (Enum<? extends T> v : e)

          { sb.append(sep); sb.append(v.name()); sep = "|"; }
          return sb.toString();
          }
          */
          private String getEnumValues(Enum[] e) {
          StringBuilder sb = new StringBuilder();
          String sep = "";
          for (Enum v : e) { sb.append(sep); sb.append(v.name()); sep = "|"; }

          return sb.toString();
          }

          next first the ivy build.xml and build-contrib scripts are broken,
          as they fail to set the correct compiler javac.target=1.7 everywhere.

          modify all of these to include the following in all javac targets:

          build-contrib.xml:

          <property name="javac.debug" value="on"/>
          <property name="javac.version" value="1.7"/>

          ...

          <!-- ====================================================== -->
          <!-- Compile a Hadoop contrib's files -->
          <!-- ====================================================== -->
          <target name="compile" depends="init, ivy-retrieve-common" unless="skip.contrib">
          <echo message="contrib: $

          {name}

          "/>
          <javac
          encoding="$

          {build.encoding}

          "
          srcdir="$

          {src.dir}

          "
          includes="*/.java"
          destdir="$

          {build.classes}

          "
          target="$

          {javac.version}"
          source="${javac.version}

          "
          optimize="$

          {javac.optimize}

          "
          debug="$

          {javac.debug}

          "
          deprecation="$

          {javac.deprecation}

          ">
          <classpath refid="contrib-classpath"/>
          </javac>
          </target>

          Next we fix the hadoop utilities Shell.java to use cygwin paths:

          Shell.java:

          /** Set to true on Windows platforms */
          public static final boolean WINDOWS /* borrowed from Path.WINDOWS */
          = System.getProperty("os.name").startsWith("Windows");

          /** a Unix command to get the current user's name */
          public final static String USER_NAME_COMMAND = (WINDOWS ? "/cygwin/bin/whoami" : "whoami");

          /** a Unix command to get the current user's groups list */
          public static String[] getGroupsCommand() {
          return new String[]

          { (WINDOWS ? "/cygwin/bin/bash" : "bash"), "-c", "groups"}

          ;
          }

          /** a Unix command to get a given user's groups list */
          public static String[] getGroupsForUserCommand(final String user) {
          //'groups username' command return is non-consistent across different unixes
          return new String []

          {(WINDOWS ? "/cygwin/bin/bash" : "bash"), "-c", "id -Gn " + user}

          ;
          }

          /** a Unix command to get a given netgroup's user list */
          public static String[] getUsersForNetgroupCommand(final String netgroup) {
          //'groups username' command return is non-consistent across different unixes
          return new String []

          {(WINDOWS ? "/cygwin/bin/bash" : "bash"), "-c", "getent netgroup " + netgroup}

          ;
          }

          /** Return a Unix command to get permission information. */
          public static String[] getGET_PERMISSION_COMMAND() {
          //force /bin/ls, except on windows.
          return new String[]

          {(WINDOWS ? "/cygwin/bin/ls" : "/bin/ls"), "-ld"}

          ;
          }

          /** a Unix command to set permission */
          public static final String SET_PERMISSION_COMMAND = (WINDOWS ? "/cygwin/bin/chmod" : "chmod");

          /** a Unix command to set owner */
          public static final String SET_OWNER_COMMAND = (WINDOWS ? "/cygwin/bin/chown" : "chown");

          /** a Unix command to set group */
          public static final String SET_GROUP_COMMAND = (WINDOWS ? "/cygwin/bin/chgrp" : "chgrp");

          /** a Unix command to get ulimit of a process. */
          public static final String ULIMIT_COMMAND = "ulimit";


          Lastly and despite this fix, hadoop filesystem's FileUtil complains
          about RawLocalFileSystem, breaking during the directory creation and
          verification because the shell's return value is improperly parsed.

          You can fix this in a number of ways. I took the lazy approach and
          just made all mkdir functions catch all IOExceptions silently.

          RawLocalFileSystem.java:

          /**

          • Creates the specified directory hierarchy. Does not
          • treat existence as an error.
            */
            public boolean mkdirs(Path f) throws IOException {
            boolean b = false;
            try { Path parent = f.getParent(); File p2f = pathToFile(f); b = (parent == null || mkdirs(parent)) && (p2f.mkdir() || p2f.isDirectory()); }

            catch (IOException e) {}
            return b;
            }

          /**

          {@inheritDoc}

          */
          @Override
          public boolean mkdirs(Path f, FsPermission permission) throws IOException {
          boolean b = false;
          try

          { b = mkdirs(f); setPermission(f, permission); }

          catch (IOException e) {}
          return b;
          }

          Finally, rebuild hadoop with "ant -f build.xml compile".
          copy the jars in the build directory oevrwriting the
          existing jars in the hadoop home parent directory.

          reformat the namenode.

          and run start-all.sh.

          you should see 4 java processes for the namenode, datanode,
          jobtracker, and tasktracker. that was a lot of yak shaving
          just to get this running.

          Show
          FKorning added a comment - There's a bunch of issues at work. I've patched this up locally on my own 1.0.2-SNAPSHOT, but it takes a lot of yak-shaving to fix. — First you need to set up hadoop-1.0.1 including source, ant, ivy, and cygwin with ssh/ssl and tcp_wrappers. Then use sshd_config to create a cyg_server priviledged user. From an admin cygwin shell, you then have to edit the /etc/passwd file and give that user a valid shell and user home, change the password for the user, and finally generate ssh keys for the user and copy the user's id_rsa.pub public key into ~/.ssh/authorized_keys. if done right you should be able to ssh cyg_server@localhost. — Now the main problem is a confusion between the hadoop shell scripts that expect unix paths like /tmp, and the haddop java binaries who interpret this path as C:\tmp. Unfortunately, neither Cygwin symlinks nor even Windows NT Junctions are supported by the java io filesystem. Thus the only way to get around this is to enforce the cygwin paths to be identical to windows paths. I get around this by creating a circular symlink in "/cygwin" -> "/". To avoid confusion with "C:" drive mappings, all my paths are relative. This means that windows "\cygwin\tmp" equals cygwin's "/cygwin/tmp". For pid files use /cygwin/tmp/ For tmp file use /cygwin/tmp/haddop-$ {USER}/ For log files use /cygwin/tmp/haddop-${USER} /logs/ — First the ssh slaves invocation warpper is broken because it fails to provide the user's ssh login, which isn't defaulted to in cygwin openssh. slaves.sh: for slave in `cat "$HOSTLIST"|sed "s/#.*$//;/^$/d"`; do ssh -l $USER $HADOOP_SSH_OPTS $slave $"$ {@// /\\ } " \ 2>&1 | sed "s/^/$slave: /" & if [ "$HADOOP_SLAVE_SLEEP" != "" ]; then sleep $HADOOP_SLAVE_SLEEP fi done Next the hadoop shell scripts are broken. you need to fix the environments for cygwin paths in hadoop-env.sh, and then make sure this file is invoked by both hadoop-config.sh, and finally the hadoop* sh wrapper script. For me its JRE java invocation was also broken, so I provide the whole srcript below. hadoop-env.sh: HADOOP_PID_DIR=/cygwin/tmp/ HADOOP_TMP_DIR=/cygwin/tmp/hadoop-$ {USER} HADOOP_LOG_DIR=/cygwin/tmp/hadoop-${USER} /logs hadoop (sh): #!/usr/bin/env bash Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at # http://www.apache.org/licenses/LICENSE-2.0 # Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. The Hadoop command script # Environment Variables # JAVA_HOME The java implementation to use. Overrides JAVA_HOME. # HADOOP_CLASSPATH Extra Java CLASSPATH entries. # HADOOP_USER_CLASSPATH_FIRST When defined, the HADOOP_CLASSPATH is added in the beginning of the global classpath. Can be defined, for example, by doing export HADOOP_USER_CLASSPATH_FIRST=true # HADOOP_HEAPSIZE The maximum amount of heap to use, in MB. Default is 1000. # HADOOP_OPTS Extra Java runtime options. HADOOP_NAMENODE_OPTS These options are added to HADOOP_OPTS HADOOP_CLIENT_OPTS when the respective command is run. HADOOP_ {COMMAND} _OPTS etc HADOOP_JT_OPTS applies to JobTracker for e.g. HADOOP_CLIENT_OPTS applies to more than one command (fs, dfs, fsck, dfsadmin etc) # HADOOP_CONF_DIR Alternate conf dir. Default is $ {HADOOP_HOME}/conf. # # HADOOP_ROOT_LOGGER The root appender. Default is INFO,console # bin=`dirname "$0"` bin=`cd "$bin"; pwd` cygwin=false case "`uname`" in CYGWIN*) cygwin=true;; esac if [ -e "$bin"/../libexec/hadoop-config.sh ]; then . "$bin"/../libexec/hadoop-config.sh else . "$bin"/hadoop-config.sh fi # if no args specified, show usage if [ $# = 0 ]; then echo "Usage: hadoop [--config confdir] COMMAND" echo "where COMMAND is one of:" echo " namenode -format format the DFS filesystem" echo " secondarynamenode run the DFS secondary namenode" echo " namenode run the DFS namenode" echo " datanode run a DFS datanode" echo " dfsadmin run a DFS admin client" echo " mradmin run a Map-Reduce admin client" echo " fsck run a DFS filesystem checking utility" echo " fs run a generic filesystem user client" echo " balancer run a cluster balancing utility" echo " fetchdt fetch a delegation token from the NameNode" echo " jobtracker run the MapReduce job Tracker node" echo " pipes run a Pipes job" echo " tasktracker run a MapReduce task Tracker node" echo " historyserver run job history servers as a standalone daemon" echo " job manipulate MapReduce jobs" echo " queue get information regarding JobQueues" echo " version print the version" echo " jar <jar> run a jar file" echo " distcp <srcurl> <desturl> copy file or directories recursively" echo " archive -archiveName NAME -p <parent path> <src>* <dest> create a hadoop archive" echo " classpath prints the class path needed to get the" echo " Hadoop jar and the required libraries" echo " daemonlog get/set the log level for each daemon" echo " or" echo " CLASSNAME run the class named CLASSNAME" echo "Most commands print help when invoked w/o parameters." exit 1 fi # get arguments COMMAND=$1 shift # Determine if we're starting a secure datanode, and if so, redefine appropriate variables if [ "$COMMAND" == "datanode" ] && [ "$EUID" -eq 0 ] && [ -n "$HADOOP_SECURE_DN_USER" ]; then HADOOP_PID_DIR=$HADOOP_SECURE_DN_PID_DIR HADOOP_LOG_DIR=$HADOOP_SECURE_DN_LOG_DIR HADOOP_IDENT_STRING=$HADOOP_SECURE_DN_USER starting_secure_dn="true" fi if [ "$JAVA_HOME" != "" ]; then #echo "JAVA_HOME: $JAVA_HOME" JAVA_HOME="$JAVA_HOME" fi # some Java parameters if $cygwin; then JAVA_HOME=`cygpath -w "$JAVA_HOME"` #echo "cygwin JAVA_HOME: $JAVA_HOME" fi if [ "$JAVA_HOME" == "" ]; then echo "Error: JAVA_HOME is not set: $JAVA_HOME" exit 1 fi JAVA=$JAVA_HOME/bin/java JAVA_HEAP_MAX=-Xmx1000m # check envvars which might override default args if [ "$HADOOP_HEAPSIZE" != "" ]; then #echo "run with heapsize $HADOOP_HEAPSIZE" JAVA_HEAP_MAX="-Xmx""$HADOOP_HEAPSIZE""m" #echo $JAVA_HEAP_MAX fi # CLASSPATH initially contains $HADOOP_CONF_DIR CLASSPATH="${HADOOP_CONF_DIR}" if [ "$HADOOP_USER_CLASSPATH_FIRST" != "" ] && [ "$HADOOP_CLASSPATH" != "" ] ; then CLASSPATH=${CLASSPATH}:${HADOOP_CLASSPATH} fi CLASSPATH=${CLASSPATH}:$JAVA_HOME/lib/tools.jar # for developers, add Hadoop classes to CLASSPATH if [ -d "$HADOOP_HOME/build/classes" ]; then CLASSPATH=${CLASSPATH}:$HADOOP_HOME/build/classes fi if [ -d "$HADOOP_HOME/build/webapps" ]; then CLASSPATH=${CLASSPATH}:$HADOOP_HOME/build fi if [ -d "$HADOOP_HOME/build/test/classes" ]; then CLASSPATH=${CLASSPATH}:$HADOOP_HOME/build/test/classes fi if [ -d "$HADOOP_HOME/build/tools" ]; then CLASSPATH=${CLASSPATH}:$HADOOP_HOME/build/tools fi # so that filenames w/ spaces are handled correctly in loops below IFS= # for releases, add core hadoop jar & webapps to CLASSPATH if [ e $HADOOP_PREFIX/share/hadoop/hadoop-core * ]; then # binary layout if [ -d "$HADOOP_PREFIX/share/hadoop/webapps" ]; then CLASSPATH=${CLASSPATH}:$HADOOP_PREFIX/share/hadoop fi for f in $HADOOP_PREFIX/share/hadoop/hadoop-core-*.jar; do CLASSPATH=${CLASSPATH}:$f; done # add libs to CLASSPATH for f in $HADOOP_PREFIX/share/hadoop/lib/*.jar; do CLASSPATH=${CLASSPATH}:$f; done for f in $HADOOP_PREFIX/share/hadoop/lib/jsp-2.1/*.jar; do CLASSPATH=${CLASSPATH}:$f; done for f in $HADOOP_PREFIX/share/hadoop/hadoop-tools-*.jar; do TOOL_PATH=${TOOL_PATH}:$f; done else # tarball layout if [ -d "$HADOOP_HOME/webapps" ]; then CLASSPATH=${CLASSPATH}:$HADOOP_HOME fi for f in $HADOOP_HOME/hadoop-core-*.jar; do CLASSPATH=${CLASSPATH}:$f; done # add libs to CLASSPATH for f in $HADOOP_HOME/lib/*.jar; do CLASSPATH=${CLASSPATH}:$f; done if [ -d "$HADOOP_HOME/build/ivy/lib/Hadoop/common" ]; then for f in $HADOOP_HOME/build/ivy/lib/Hadoop/common/*.jar; do CLASSPATH=${CLASSPATH}:$f; done fi for f in $HADOOP_HOME/lib/jsp-2.1/*.jar; do CLASSPATH=${CLASSPATH}:$f; done for f in $HADOOP_HOME/hadoop-tools-*.jar; do TOOL_PATH=${TOOL_PATH}:$f; done for f in $HADOOP_HOME/build/hadoop-tools-*.jar; do TOOL_PATH=${TOOL_PATH}:$f; done fi # add user-specified CLASSPATH last if [ "$HADOOP_USER_CLASSPATH_FIRST" = "" ] && [ "$HADOOP_CLASSPATH" != "" ]; then CLASSPATH=${CLASSPATH}:${HADOOP_CLASSPATH} fi # default log directory & file if [ "$HADOOP_LOG_DIR" = "" ]; then HADOOP_LOG_DIR="$HADOOP_HOME/logs" fi if [ "$HADOOP_LOGFILE" = "" ]; then HADOOP_LOGFILE='hadoop.log' fi # default policy file for service-level authorization if [ "$HADOOP_POLICYFILE" = "" ]; then HADOOP_POLICYFILE="hadoop-policy.xml" fi # restore ordinary behaviour unset IFS # figure out which class to run if [ "$COMMAND" = "classpath" ] ; then if $cygwin; then CLASSPATH=`cygpath -wp "$CLASSPATH"` fi echo $CLASSPATH exit elif [ "$COMMAND" = "namenode" ] ; then CLASS='org.apache.hadoop.hdfs.server.namenode.NameNode' HADOOP_OPTS="$HADOOP_OPTS $HADOOP_NAMENODE_OPTS" elif [ "$COMMAND" = "secondarynamenode" ] ; then CLASS='org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode' HADOOP_OPTS="$HADOOP_OPTS $HADOOP_SECONDARYNAMENODE_OPTS" elif [ "$COMMAND" = "datanode" ] ; then CLASS='org.apache.hadoop.hdfs.server.datanode.DataNode' if [ "$starting_secure_dn" = "true" ]; then HADOOP_OPTS="$HADOOP_OPTS -jvm server $HADOOP_DATANODE_OPTS" else HADOOP_OPTS="$HADOOP_OPTS -server $HADOOP_DATANODE_OPTS" fi elif [ "$COMMAND" = "fs" ] ; then CLASS=org.apache.hadoop.fs.FsShell HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS" elif [ "$COMMAND" = "dfs" ] ; then CLASS=org.apache.hadoop.fs.FsShell HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS" elif [ "$COMMAND" = "dfsadmin" ] ; then CLASS=org.apache.hadoop.hdfs.tools.DFSAdmin HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS" elif [ "$COMMAND" = "mradmin" ] ; then CLASS=org.apache.hadoop.mapred.tools.MRAdmin HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS" elif [ "$COMMAND" = "fsck" ] ; then CLASS=org.apache.hadoop.hdfs.tools.DFSck HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS" elif [ "$COMMAND" = "balancer" ] ; then CLASS=org.apache.hadoop.hdfs.server.balancer.Balancer HADOOP_OPTS="$HADOOP_OPTS $HADOOP_BALANCER_OPTS" elif [ "$COMMAND" = "fetchdt" ] ; then CLASS=org.apache.hadoop.hdfs.tools.DelegationTokenFetcher elif [ "$COMMAND" = "jobtracker" ] ; then CLASS=org.apache.hadoop.mapred.JobTracker HADOOP_OPTS="$HADOOP_OPTS $HADOOP_JOBTRACKER_OPTS" elif [ "$COMMAND" = "historyserver" ] ; then CLASS=org.apache.hadoop.mapred.JobHistoryServer HADOOP_OPTS="$HADOOP_OPTS $HADOOP_JOB_HISTORYSERVER_OPTS" elif [ "$COMMAND" = "tasktracker" ] ; then CLASS=org.apache.hadoop.mapred.TaskTracker HADOOP_OPTS="$HADOOP_OPTS $HADOOP_TASKTRACKER_OPTS" elif [ "$COMMAND" = "job" ] ; then CLASS=org.apache.hadoop.mapred.JobClient HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS" elif [ "$COMMAND" = "queue" ] ; then CLASS=org.apache.hadoop.mapred.JobQueueClient HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS" elif [ "$COMMAND" = "pipes" ] ; then CLASS=org.apache.hadoop.mapred.pipes.Submitter HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS" elif [ "$COMMAND" = "version" ] ; then CLASS=org.apache.hadoop.util.VersionInfo HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS" elif [ "$COMMAND" = "jar" ] ; then CLASS=org.apache.hadoop.util.RunJar HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS" elif [ "$COMMAND" = "distcp" ] ; then CLASS=org.apache.hadoop.tools.DistCp CLASSPATH=${CLASSPATH}:${TOOL_PATH} HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS" elif [ "$COMMAND" = "daemonlog" ] ; then CLASS=org.apache.hadoop.log.LogLevel HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS" elif [ "$COMMAND" = "archive" ] ; then CLASS=org.apache.hadoop.tools.HadoopArchives CLASSPATH=${CLASSPATH}:${TOOL_PATH} HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS" elif [ "$COMMAND" = "sampler" ] ; then CLASS=org.apache.hadoop.mapred.lib.InputSampler HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS" else CLASS=$COMMAND fi # cygwin path translation if $cygwin; then JAVA_HOME=`cygpath -w "$JAVA_HOME"` CLASSPATH=`cygpath -wp "$CLASSPATH"` HADOOP_HOME=`cygpath -w "$HADOOP_HOME"` HADOOP_LOG_DIR=`cygpath -w "$HADOOP_LOG_DIR"` TOOL_PATH=`cygpath -wp "$TOOL_PATH"` fi # setup 'java.library.path' for native-hadoop code if necessary JAVA_LIBRARY_PATH='' if [ -d "${HADOOP_HOME} /build/native" -o -d "$ {HADOOP_HOME}/lib/native" -o -e "${HADOOP_PREFIX}/lib/libhadoop.a" ]; then JAVA_PLATFORM=`${JAVA} -classpath ${CLASSPATH} -Xmx32m ${HADOOP_JAVA_PLATFORM_OPTS} org.apache.hadoop.util.PlatformName | sed -e "s/ /_/g"` #echo "JAVA_PLATFORM: $JAVA_PLATFORM" if [ "$JAVA_PLATFORM" = "Windows_7-amd64-64" ]; then JSVC_ARCH="amd64" elif [ "$JAVA_PLATFORM" = "Linux-amd64-64" ]; then JSVC_ARCH="amd64" else JSVC_ARCH="i386" fi if [ -d "$HADOOP_HOME/build/native" ]; then JAVA_LIBRARY_PATH=${HADOOP_HOME} /build/native/$ {JAVA_PLATFORM}/lib fi if [ -d "${HADOOP_HOME}/lib/native" ]; then if [ "x$JAVA_LIBRARY_PATH" != "x" ]; then JAVA_LIBRARY_PATH=${JAVA_LIBRARY_PATH}:${HADOOP_HOME}/lib/native/${JAVA_PLATFORM} else JAVA_LIBRARY_PATH=$ {HADOOP_HOME} /lib/native/$ {JAVA_PLATFORM} fi fi if [ -e "$ {HADOOP_PREFIX}/lib/libhadoop.a" ]; then JAVA_LIBRARY_PATH=${HADOOP_PREFIX} /lib fi fi cygwin path translation if $cygwin; then JAVA_LIBRARY_PATH=`cygpath -wp "$JAVA_LIBRARY_PATH"` PATH="/cygwin/bin:/cygwin/usr/bin:`cygpath -p $ {PATH} `" fi HADOOP_OPTS="$HADOOP_OPTS -Dhadoop.tmp.dir=$HADOOP_TMP_DIR" HADOOP_OPTS="$HADOOP_OPTS -Dhadoop.log.dir=$HADOOP_LOG_DIR" HADOOP_OPTS="$HADOOP_OPTS -Dhadoop.log.file=$HADOOP_LOGFILE" HADOOP_OPTS="$HADOOP_OPTS -Dhadoop.home.dir=$HADOOP_HOME" HADOOP_OPTS="$HADOOP_OPTS -Dhadoop.id.str=$HADOOP_IDENT_STRING" HADOOP_OPTS="$HADOOP_OPTS -Dhadoop.root.logger=$ {HADOOP_ROOT_LOGGER:-INFO,console} " #turn security logger on the namenode and jobtracker only if [ $COMMAND = "namenode" ] || [ $COMMAND = "jobtracker" ]; then HADOOP_OPTS="$HADOOP_OPTS -Dhadoop.security.logger=$ {HADOOP_SECURITY_LOGGER:-INFO,DRFAS} " else HADOOP_OPTS="$HADOOP_OPTS -Dhadoop.security.logger=$ {HADOOP_SECURITY_LOGGER:-INFO,NullAppender} " fi if [ "x$JAVA_LIBRARY_PATH" != "x" ]; then HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=$JAVA_LIBRARY_PATH" fi HADOOP_OPTS="$HADOOP_OPTS -Dhadoop.policy.file=$HADOOP_POLICYFILE" Check to see if we should start a secure datanode if [ "$starting_secure_dn" = "true" ]; then if [ "$HADOOP_PID_DIR" = "" ]; then HADOOP_SECURE_DN_PID="/tmp/hadoop_secure_dn.pid" else HADOOP_SECURE_DN_PID="$HADOOP_PID_DIR/hadoop_secure_dn.pid" fi exec "$HADOOP_HOME/libexec/jsvc.$ {JSVC_ARCH} " -Dproc_$COMMAND -outfile "$HADOOP_LOG_DIR/jsvc.out" \ -errfile "$HADOOP_LOG_DIR/jsvc.err" \ -pidfile "$HADOOP_SECURE_DN_PID" \ -nodetach \ -user "$HADOOP_SECURE_DN_USER" \ -cp "$CLASSPATH" \ $JAVA_HEAP_MAX $HADOOP_OPTS \ org.apache.hadoop.hdfs.server.datanode.SecureDataNodeStarter "$@" else run it exec "$JAVA" -Dproc_$COMMAND $JAVA_HEAP_MAX $HADOOP_OPTS -classpath "$CLASSPATH" $CLASS "$@" fi Next the hadoop fs and utilities are broken, as they expect shells with POSIX /bin executables in their path (bash,chmod,chown,chgrp) For various reasons it's a real bad idea to add "/cygwin/bin" to your windows path, so we're going to have to fix the utility classes to be cygwin aware and use the "/cygwin/bin" binaries instead. This is why you need the source, because we're going to have to fix the java source and recompile the hadoop core libraries (and why you need ant ivy). Before we do this, the contrib Gridmix is broken as it uses a strange generic Enum code that just craps out in jdk/jre 1.7 and above. The fix is to dumb it down and use untyped Enums. Gridmix.java: /* private <T> String getEnumValues(Enum<? extends T>[] e) { StringBuilder sb = new StringBuilder(); String sep = ""; for (Enum<? extends T> v : e) { sb.append(sep); sb.append(v.name()); sep = "|"; } return sb.toString(); } */ private String getEnumValues(Enum[] e) { StringBuilder sb = new StringBuilder(); String sep = ""; for (Enum v : e) { sb.append(sep); sb.append(v.name()); sep = "|"; } return sb.toString(); } — next first the ivy build.xml and build-contrib scripts are broken, as they fail to set the correct compiler javac.target=1.7 everywhere. modify all of these to include the following in all javac targets: build-contrib.xml: <property name="javac.debug" value="on"/> <property name="javac.version" value="1.7"/> ... <!-- ====================================================== --> <!-- Compile a Hadoop contrib's files --> <!-- ====================================================== --> <target name="compile" depends="init, ivy-retrieve-common" unless="skip.contrib"> <echo message="contrib: $ {name} "/> <javac encoding="$ {build.encoding} " srcdir="$ {src.dir} " includes="* / .java" destdir="$ {build.classes} " target="$ {javac.version}" source="${javac.version} " optimize="$ {javac.optimize} " debug="$ {javac.debug} " deprecation="$ {javac.deprecation} "> <classpath refid="contrib-classpath"/> </javac> </target> — Next we fix the hadoop utilities Shell.java to use cygwin paths: Shell.java: /** Set to true on Windows platforms */ public static final boolean WINDOWS /* borrowed from Path.WINDOWS */ = System.getProperty("os.name").startsWith("Windows"); /** a Unix command to get the current user's name */ public final static String USER_NAME_COMMAND = (WINDOWS ? "/cygwin/bin/whoami" : "whoami"); /** a Unix command to get the current user's groups list */ public static String[] getGroupsCommand() { return new String[] { (WINDOWS ? "/cygwin/bin/bash" : "bash"), "-c", "groups"} ; } /** a Unix command to get a given user's groups list */ public static String[] getGroupsForUserCommand(final String user) { //'groups username' command return is non-consistent across different unixes return new String [] {(WINDOWS ? "/cygwin/bin/bash" : "bash"), "-c", "id -Gn " + user} ; } /** a Unix command to get a given netgroup's user list */ public static String[] getUsersForNetgroupCommand(final String netgroup) { //'groups username' command return is non-consistent across different unixes return new String [] {(WINDOWS ? "/cygwin/bin/bash" : "bash"), "-c", "getent netgroup " + netgroup} ; } /** Return a Unix command to get permission information. */ public static String[] getGET_PERMISSION_COMMAND() { //force /bin/ls, except on windows. return new String[] {(WINDOWS ? "/cygwin/bin/ls" : "/bin/ls"), "-ld"} ; } /** a Unix command to set permission */ public static final String SET_PERMISSION_COMMAND = (WINDOWS ? "/cygwin/bin/chmod" : "chmod"); /** a Unix command to set owner */ public static final String SET_OWNER_COMMAND = (WINDOWS ? "/cygwin/bin/chown" : "chown"); /** a Unix command to set group */ public static final String SET_GROUP_COMMAND = (WINDOWS ? "/cygwin/bin/chgrp" : "chgrp"); /** a Unix command to get ulimit of a process. */ public static final String ULIMIT_COMMAND = "ulimit"; Lastly and despite this fix, hadoop filesystem's FileUtil complains about RawLocalFileSystem, breaking during the directory creation and verification because the shell's return value is improperly parsed. You can fix this in a number of ways. I took the lazy approach and just made all mkdir functions catch all IOExceptions silently. RawLocalFileSystem.java: /** Creates the specified directory hierarchy. Does not treat existence as an error. */ public boolean mkdirs(Path f) throws IOException { boolean b = false; try { Path parent = f.getParent(); File p2f = pathToFile(f); b = (parent == null || mkdirs(parent)) && (p2f.mkdir() || p2f.isDirectory()); } catch (IOException e) {} return b; } /** {@inheritDoc} */ @Override public boolean mkdirs(Path f, FsPermission permission) throws IOException { boolean b = false; try { b = mkdirs(f); setPermission(f, permission); } catch (IOException e) {} return b; } — Finally, rebuild hadoop with "ant -f build.xml compile". copy the jars in the build directory oevrwriting the existing jars in the hadoop home parent directory. reformat the namenode. and run start-all.sh. you should see 4 java processes for the namenode, datanode, jobtracker, and tasktracker. that was a lot of yak shaving just to get this running.
          Hide
          Dave Latham added a comment -

          Here's my understanding of this issue from digging around a bit in case it's helpful for others. I could be mistaken on some of the details.

          Some time ago, RawLocalFileSystem.setPermission used to use a shell exec command to fork a process to alter permissions of a file.

          Somewhere along the 0.20 branch (I believe in the 0.20.security branch) it was decided that this was too slow, and instead java.io.File.set

          {Readable|Writable|Executable}

          should be used (see HADOOP-6304 for one such issue. It has a patch with the logic that wound up in FileUtil though perhaps committed from a different patch). However the java.io.File methods don't allow one to directly set all the permissions so the code first has to clear permissions, then build them up again. This resulted in two problems. First, there is a race condition when the file briefly has no permissions even for the owner (see MAPREDUCE-2238 for more detail). Second, Windows doesn't support clearing all permissions on the file (this JIRA).

          The first problem was worked around in HADOOP-7110 (HADOOP-7432 backported it to this branch) by using JNI native code instead. However, if the native code is not available, then it falls back to the java.io.File methods. So, the second problem still remains, that FileUtil.setPermission (and thus the RawLocalFileSystem setPermission) does not work on Windows because Windows does not have the native code implementation and also fails the java.io.File fallback.

          The issues FKorning ran into in a comment above appear to be wider than this particular JIRA, though I may have misunderstood what led to his shorn yak.

          Windows is listed as a supported platform for Hadoop, and some of our developers use Windows as a development environment, so it's important for us that hadoop at least functions on Windows, even if it's not as performant as our production clusters on Linux. I noted that this is currently the highest voted open JIRA for hadoop.

          In order for it to function on Windows, I added a final fallback in FileUtil.setPermission to revert to the older behavior of using a shell exec fork for setPermission when the other methods fail. Perhaps it will be helpful for others:

            public static void setPermission(File f, FsPermission permission
                                             ) throws IOException {
              FsAction user = permission.getUserAction();
              FsAction group = permission.getGroupAction();
              FsAction other = permission.getOtherAction();
          
              // use the native/fork if the group/other permissions are different
              // or if the native is available    
              if (group != other || NativeIO.isAvailable()) {
                execSetPermission(f, permission);
                return;
              }
              
              try
              {
          	    boolean rv = true;
          	    
          	    // read perms
          	    rv = f.setReadable(group.implies(FsAction.READ), false);
          	    checkReturnValue(rv, f, permission);
          	    if (group.implies(FsAction.READ) != user.implies(FsAction.READ)) {
          	      f.setReadable(user.implies(FsAction.READ), true);
          	      checkReturnValue(rv, f, permission);
          	    }
          	
          	    // write perms
          	    rv = f.setWritable(group.implies(FsAction.WRITE), false);
          	    checkReturnValue(rv, f, permission);
          	    if (group.implies(FsAction.WRITE) != user.implies(FsAction.WRITE)) {
          	      f.setWritable(user.implies(FsAction.WRITE), true);
          	      checkReturnValue(rv, f, permission);
          	    }
          	
          	    // exec perms
          	    rv = f.setExecutable(group.implies(FsAction.EXECUTE), false);
          	    checkReturnValue(rv, f, permission);
          	    if (group.implies(FsAction.EXECUTE) != user.implies(FsAction.EXECUTE)) {
          	      f.setExecutable(user.implies(FsAction.EXECUTE), true);
          	      checkReturnValue(rv, f, permission);
          	    }
              }
              catch (IOException ioe)
              {
              	LOG.warn("Java file permissions failed to set " + f + " to " + permission + " falling back to fork");
              	execSetPermission(f, permission);
              }
              
            }
          
          Show
          Dave Latham added a comment - Here's my understanding of this issue from digging around a bit in case it's helpful for others. I could be mistaken on some of the details. Some time ago, RawLocalFileSystem.setPermission used to use a shell exec command to fork a process to alter permissions of a file. Somewhere along the 0.20 branch (I believe in the 0.20.security branch) it was decided that this was too slow, and instead java.io.File.set {Readable|Writable|Executable} should be used (see HADOOP-6304 for one such issue. It has a patch with the logic that wound up in FileUtil though perhaps committed from a different patch). However the java.io.File methods don't allow one to directly set all the permissions so the code first has to clear permissions, then build them up again. This resulted in two problems. First, there is a race condition when the file briefly has no permissions even for the owner (see MAPREDUCE-2238 for more detail). Second, Windows doesn't support clearing all permissions on the file (this JIRA). The first problem was worked around in HADOOP-7110 ( HADOOP-7432 backported it to this branch) by using JNI native code instead. However, if the native code is not available, then it falls back to the java.io.File methods. So, the second problem still remains, that FileUtil.setPermission (and thus the RawLocalFileSystem setPermission) does not work on Windows because Windows does not have the native code implementation and also fails the java.io.File fallback. The issues FKorning ran into in a comment above appear to be wider than this particular JIRA, though I may have misunderstood what led to his shorn yak. Windows is listed as a supported platform for Hadoop, and some of our developers use Windows as a development environment, so it's important for us that hadoop at least functions on Windows, even if it's not as performant as our production clusters on Linux. I noted that this is currently the highest voted open JIRA for hadoop. In order for it to function on Windows, I added a final fallback in FileUtil.setPermission to revert to the older behavior of using a shell exec fork for setPermission when the other methods fail. Perhaps it will be helpful for others: public static void setPermission(File f, FsPermission permission ) throws IOException { FsAction user = permission.getUserAction(); FsAction group = permission.getGroupAction(); FsAction other = permission.getOtherAction(); // use the native /fork if the group/other permissions are different // or if the native is available if (group != other || NativeIO.isAvailable()) { execSetPermission(f, permission); return ; } try { boolean rv = true ; // read perms rv = f.setReadable(group.implies(FsAction.READ), false ); checkReturnValue(rv, f, permission); if (group.implies(FsAction.READ) != user.implies(FsAction.READ)) { f.setReadable(user.implies(FsAction.READ), true ); checkReturnValue(rv, f, permission); } // write perms rv = f.setWritable(group.implies(FsAction.WRITE), false ); checkReturnValue(rv, f, permission); if (group.implies(FsAction.WRITE) != user.implies(FsAction.WRITE)) { f.setWritable(user.implies(FsAction.WRITE), true ); checkReturnValue(rv, f, permission); } // exec perms rv = f.setExecutable(group.implies(FsAction.EXECUTE), false ); checkReturnValue(rv, f, permission); if (group.implies(FsAction.EXECUTE) != user.implies(FsAction.EXECUTE)) { f.setExecutable(user.implies(FsAction.EXECUTE), true ); checkReturnValue(rv, f, permission); } } catch (IOException ioe) { LOG.warn( "Java file permissions failed to set " + f + " to " + permission + " falling back to fork" ); execSetPermission(f, permission); } }
          Hide
          FKorning added a comment -

          I've managed to get this working to the point where jobs are dispatched, tasks executed, and results compiled.

          However we still need to get the servlets to understand cygwin symlinks. I have no idea how to do this in Jetty.

          These two links show how to allow Tomcat and jetty to follow symlinks, but I don't know if this works in cygwin.

          Otherwise we'll have to open up the jetty code and replace java.io.File with org.apache.hadoop.fs.LinkedFile.

          Show
          FKorning added a comment - I've managed to get this working to the point where jobs are dispatched, tasks executed, and results compiled. http://en.wikisource.org/wiki/User:Fkorning/Code/Haddoop-on-Cygwin However we still need to get the servlets to understand cygwin symlinks. I have no idea how to do this in Jetty. These two links show how to allow Tomcat and jetty to follow symlinks, but I don't know if this works in cygwin. http://www.lamoree.com/machblog/index.cfm?event=showEntry&entryId=A2F0ED76-A500-41A6-A1DFDE0D1996F925 http://stackoverflow.com/questions/315093/configure-symlinks-for-single-directory-in-tomcat Otherwise we'll have to open up the jetty code and replace java.io.File with org.apache.hadoop.fs.LinkedFile.
          Show
          FKorning added a comment - make that: http://en.wikisource.org/wiki/User:Fkorning/Code/Hadoop-on-Cygwin
          Hide
          Ismail Shaik added a comment -

          I have tested with v1.0.1 and same problen within eclipse/cygwin.Do you have quick workaround?

          Show
          Ismail Shaik added a comment - I have tested with v1.0.1 and same problen within eclipse/cygwin.Do you have quick workaround?
          Hide
          FKorning added a comment -

          Ismail,

          You misunderstand, I haven't patched the offical 1.0.1 codebase:
          I'm not an official hadoop contributor, I'm not really sure if the
          developers will approve all my fixes such as adding a java.io.File
          wrapper etc.

          (to wit, no persons from the hadoop team, not even those who were
          assigned to the various cygwin bugs, have contacted me on this).

          what I provide are instructions for you to patch 1.0.1 yourself,
          and hopefully provide guidance to the official hadoop developpers
          on porting the software properly to cygwin.

          and no, if you read the doc, you'll see why there is no quick
          workaround, short of falling back to 0.20. But I rather think
          most people would want a 1.x, especially compiled with 64-bit.

          it's not a simple code fix. you need to understand how cygwin works;
          you need to customize and configure cygwin and the pre-requisite
          sofware such as java, ant, ivy, maven; you need to patch scripts;
          you need to patch code; you need to add brand new classes to wrap
          java.io.File; you need to rebuild, reconfigure and redeploy it;

          for a complete working fix, I want to port the latest version
          and release the code on sourceforge, but I'm a bit busy now.

          Show
          FKorning added a comment - Ismail, You misunderstand, I haven't patched the offical 1.0.1 codebase: I'm not an official hadoop contributor, I'm not really sure if the developers will approve all my fixes such as adding a java.io.File wrapper etc. (to wit, no persons from the hadoop team, not even those who were assigned to the various cygwin bugs, have contacted me on this). what I provide are instructions for you to patch 1.0.1 yourself, and hopefully provide guidance to the official hadoop developpers on porting the software properly to cygwin. and no, if you read the doc, you'll see why there is no quick workaround, short of falling back to 0.20. But I rather think most people would want a 1.x, especially compiled with 64-bit. it's not a simple code fix. you need to understand how cygwin works; you need to customize and configure cygwin and the pre-requisite sofware such as java, ant, ivy, maven; you need to patch scripts; you need to patch code; you need to add brand new classes to wrap java.io.File; you need to rebuild, reconfigure and redeploy it; for a complete working fix, I want to port the latest version and release the code on sourceforge, but I'm a bit busy now.
          Hide
          Joshua Caplan added a comment -

          May I suggest a simple workaround for Windows users struggling with this new impediment.

          create a WinLocalFileSystem class (subclass of LocalFileSystem) which ignores IOExceptions on setPermissions() or, if you're feeling ambitious, does something more appropriate when trying to set them. You will also need to override mkdirs( Path, FsPermission ) to delegate to mkdirs( Path ) and call this.setPermission.

          Register this class as the implementation of the "file" schema, using the property definition

          fs.file.impl=com.package.name.WinLocalFileSystem

          That got me past it for local testing.

          Show
          Joshua Caplan added a comment - May I suggest a simple workaround for Windows users struggling with this new impediment. create a WinLocalFileSystem class (subclass of LocalFileSystem) which ignores IOExceptions on setPermissions() or, if you're feeling ambitious, does something more appropriate when trying to set them. You will also need to override mkdirs( Path, FsPermission ) to delegate to mkdirs( Path ) and call this.setPermission. Register this class as the implementation of the "file" schema, using the property definition fs.file.impl=com.package.name.WinLocalFileSystem That got me past it for local testing.
          Hide
          Todd Fast added a comment -

          Thanks, Joshua! Your workaround got us running locally on Hadoop 1.0.3 without any issues. Here is a GitHub repository for the source (as well as a pre-built JAR):

          https://github.com/congainc/patch-hadoop_7682-1.0.x-win

          Show
          Todd Fast added a comment - Thanks, Joshua! Your workaround got us running locally on Hadoop 1.0.3 without any issues. Here is a GitHub repository for the source (as well as a pre-built JAR): https://github.com/congainc/patch-hadoop_7682-1.0.x-win
          Hide
          丁冬超 added a comment -

          the same on hadoop 1.0.1 with win7 and cygwin 。
          because just test ,so i suggest a easy workaround :
          1、 alert FileUtil
          mkdir $HADOOP_HOME/classes
          and then alter the org.apache.hadoop.fs.FileUtil.checkReturnValue remove the Exception .
          build 、copy FileUtil.class to $HADOOP_HOME/classes 。

          2 、alter classpath

          add shell in $HADOOP_HOME/hadoop file :

          if [ -d "$HADOOP_HOME/classes" ]; then
          CLASSPATH=$

          {CLASSPATH}

          :$HADOOP_HOME/classes
          fi

          make sure this classpath is in front of $HADOOP_HOME/hadoop-core-1...jar 。

          restart OK

          Show
          丁冬超 added a comment - the same on hadoop 1.0.1 with win7 and cygwin 。 because just test ,so i suggest a easy workaround : 1、 alert FileUtil mkdir $HADOOP_HOME/classes and then alter the org.apache.hadoop.fs.FileUtil.checkReturnValue remove the Exception . build 、copy FileUtil.class to $HADOOP_HOME/classes 。 2 、alter classpath add shell in $HADOOP_HOME/hadoop file : if [ -d "$HADOOP_HOME/classes" ]; then CLASSPATH=$ {CLASSPATH} :$HADOOP_HOME/classes fi make sure this classpath is in front of $HADOOP_HOME/hadoop-core-1. . .jar 。 restart OK
          Hide
          visionersadak added a comment -

          thanks a ton todd the path u mentioned got our task tracker running on windows but when we started creating a HAR job its throwing this error

          java.io.IOException: Failed to set permissions of path: \tmp\hadoop-cyg_server\mapred\local\taskTracker\sadak(myusername) to 0700

          Show
          visionersadak added a comment - thanks a ton todd the path u mentioned got our task tracker running on windows but when we started creating a HAR job its throwing this error java.io.IOException: Failed to set permissions of path: \tmp\hadoop-cyg_server\mapred\local\taskTracker\sadak(myusername) to 0700
          Hide
          Luiz Veronesi added a comment -

          Thanks Todd Fast for this workaround.

          That also solved running Nutch within Eclipse.

          Exactly the same thing must be done to make it works.

          Show
          Luiz Veronesi added a comment - Thanks Todd Fast for this workaround. That also solved running Nutch within Eclipse. Exactly the same thing must be done to make it works.
          Hide
          frank added a comment -

          Anyone find out what is causing visionersadak's error? Todd's fix worked great; now I'm getting the same "failed to set permissions" error.

          Show
          frank added a comment - Anyone find out what is causing visionersadak's error? Todd's fix worked great; now I'm getting the same "failed to set permissions" error.
          Hide
          Todd Fast added a comment -

          It sounded as if he was running hadoop in pseudo-distributed or fully distributed mode, and perhaps the way the child processes are spawned doesn't include the patch in the classpath. A quick test to verify this theory, though I wouldn't do this under normal running conditions, would be to put the patch JAR in the JDK's $

          {JAVA_HOME}

          /jre/lib/ext directory, which includes it in the classpath for all VMs. If that works, then you know that it's something to do with the local classpath when the hadoop VMs are started. At that point, you can further debug how the processes are spawned, or you can run in standalone mode instead.

          For standalone mode, the patch is automatically added to the classpath from the $

          {HADOOP_HOME}

          /lib directory, and if the patch is installed correctly, you should see some logging that indicates it is working.

          Show
          Todd Fast added a comment - It sounded as if he was running hadoop in pseudo-distributed or fully distributed mode, and perhaps the way the child processes are spawned doesn't include the patch in the classpath. A quick test to verify this theory, though I wouldn't do this under normal running conditions, would be to put the patch JAR in the JDK's $ {JAVA_HOME} /jre/lib/ext directory, which includes it in the classpath for all VMs. If that works, then you know that it's something to do with the local classpath when the hadoop VMs are started. At that point, you can further debug how the processes are spawned, or you can run in standalone mode instead. For standalone mode, the patch is automatically added to the classpath from the $ {HADOOP_HOME} /lib directory, and if the patch is installed correctly, you should see some logging that indicates it is working.
          Hide
          Han Hui Wen added a comment -

          I find one approach :

          Just need change file $

          {HADOOP_HOME}

          /bin/hadoop-config.sh

          from :

          JAVA_PLATFORM=`CLASSPATH=$

          {CLASSPATH} ${JAVA} -Xmx32m ${HADOOP_JAVA_PLATFORM_OPTS} org.apache.hadoop.util.PlatformName | sed -e ”s/ /_/g”`

          to:

          JAVA_PLATFORM=`CLASSPATH=${CLASSPATH}

          $

          {JAVA}

          -Xmx32m -classpath $

          {HADOOP_COMMON_HOME}

          /hadoop-common-0.21.0.jar org.apache.hadoop.util.PlatformName | sed -e ”s/ /_/g”`

          Show
          Han Hui Wen added a comment - I find one approach : Just need change file $ {HADOOP_HOME} /bin/hadoop-config.sh from : JAVA_PLATFORM=`CLASSPATH=$ {CLASSPATH} ${JAVA} -Xmx32m ${HADOOP_JAVA_PLATFORM_OPTS} org.apache.hadoop.util.PlatformName | sed -e ”s/ /_/g”` to: JAVA_PLATFORM=`CLASSPATH=${CLASSPATH} $ {JAVA} -Xmx32m -classpath $ {HADOOP_COMMON_HOME} /hadoop-common-0.21.0.jar org.apache.hadoop.util.PlatformName | sed -e ”s/ /_/g”`
          Hide
          Kirill Vergun added a comment -

          visionersadak,

          are you running your installation under “single node” (pseudo-distributed) configuration?
          I am trying to do my own patching, it may work on such type of configuration.

          https://github.com/o-nix/hadoop-patches

          But it slows down every hadoop shell command execution a lot.

          Show
          Kirill Vergun added a comment - visionersadak, are you running your installation under “single node” (pseudo-distributed) configuration? I am trying to do my own patching, it may work on such type of configuration. https://github.com/o-nix/hadoop-patches But it slows down every hadoop shell command execution a lot.
          Hide
          FKorning added a comment -

          yes. it is extremely slow, which sort of makes the whole windows thing a bit moot. then again, if you have a farm of windows boxes sitting idle, you may as well use their cycles...

          Sent from my iPhone

          Show
          FKorning added a comment - yes. it is extremely slow, which sort of makes the whole windows thing a bit moot. then again, if you have a farm of windows boxes sitting idle, you may as well use their cycles... Sent from my iPhone
          Hide
          Daniel Czajka added a comment -

          Hey, I checked on latest stable 1.0.4/win7 and still the same error occurred. Are there any plans/timelines to fix it ?

          Cheers,
          Daniel

          Show
          Daniel Czajka added a comment - Hey, I checked on latest stable 1.0.4/win7 and still the same error occurred. Are there any plans/timelines to fix it ? Cheers, Daniel
          Hide
          Matt Giroux added a comment -

          I'm using 1.0.4...

          This issue is coming from the org.apache.hadoop.fs.FileUtil class. There is a static setPermission method that uses the standard java.io.File setPermission method which will always fail on Windows. The quickest and least destructive way to resolve this is just create three new classes:

          org.apache.hadoop.fs.Win32LocalFileSystem
          org.apache.hadoop.fs.Win32RawLocalFileSystem
          org.apache.hadoop.fs.Win32FileUtil

          They obviously extend the classes of the same name without the "Win32". Here is the code for each:

          <begin Win32FileUtil>

          package org.apache.hadoop.fs;

          import java.io.File;
          import java.io.IOException;
          import org.apache.hadoop.fs.permission.FsPermission;

          /**

          • A collection of file-processing util methods
            */
            public class Win32FileUtil extends FileUtil {

          /* override setPermission */
          public static void setPermission(File f, FsPermission permission
          ) throws IOException

          { /* assume can't set permissions on win32 */ /* do nothing */ }

          }

          <end Win32FileUtil>

          <begin Win32RawLocalFileSystem>

          package org.apache.hadoop.fs;

          import java.io.IOException;
          import org.apache.hadoop.fs.permission.FsPermission;

          /****************************************************************

          • Implement the FileSystem API for the raw local filesystem.
            *
            *****************************************************************/
            public class Win32RawLocalFileSystem extends RawLocalFileSystem {

          public Win32RawLocalFileSystem()

          { super(); }

          @Override
          public void setPermission(Path p, FsPermission permission
          ) throws IOException

          { Win32FileUtil.setPermission(pathToFile(p), permission); }

          }

          <end Win32RawLocalFileSystem>

          <begin Win32LocalFileSystem>

          package org.apache.hadoop.fs;

          /****************************************************************

          • Implement the FileSystem API for the checksumed local filesystem.
            *
            *****************************************************************/
            public class Win32LocalFileSystem extends LocalFileSystem {

          public Win32LocalFileSystem()

          { this(new Win32RawLocalFileSystem()); }

          public Win32LocalFileSystem(FileSystem rawLocalFileSystem)

          { super(rawLocalFileSystem); rfs = rawLocalFileSystem; }

          }

          <end Win32LocalFileSystem>

          For a quickest compile, just manually compile and add these classes into the hadoop-core-1.0.4.jar (with this jar on the classpath) in the same order as above (backup your jar file first to be safe). Then edit the core-site.xml configuration file with the following entry:

          <property>
          <name>fs.file.impl</name>
          <value>org.apache.hadoop.fs.Win32LocalFileSystem</value>
          </property>

          All done - should work now.

          Show
          Matt Giroux added a comment - I'm using 1.0.4... This issue is coming from the org.apache.hadoop.fs.FileUtil class. There is a static setPermission method that uses the standard java.io.File setPermission method which will always fail on Windows. The quickest and least destructive way to resolve this is just create three new classes: org.apache.hadoop.fs.Win32LocalFileSystem org.apache.hadoop.fs.Win32RawLocalFileSystem org.apache.hadoop.fs.Win32FileUtil They obviously extend the classes of the same name without the "Win32". Here is the code for each: <begin Win32FileUtil> package org.apache.hadoop.fs; import java.io.File; import java.io.IOException; import org.apache.hadoop.fs.permission.FsPermission; /** A collection of file-processing util methods */ public class Win32FileUtil extends FileUtil { /* override setPermission */ public static void setPermission(File f, FsPermission permission ) throws IOException { /* assume can't set permissions on win32 */ /* do nothing */ } } <end Win32FileUtil> <begin Win32RawLocalFileSystem> package org.apache.hadoop.fs; import java.io.IOException; import org.apache.hadoop.fs.permission.FsPermission; /**************************************************************** Implement the FileSystem API for the raw local filesystem. * *****************************************************************/ public class Win32RawLocalFileSystem extends RawLocalFileSystem { public Win32RawLocalFileSystem() { super(); } @Override public void setPermission(Path p, FsPermission permission ) throws IOException { Win32FileUtil.setPermission(pathToFile(p), permission); } } <end Win32RawLocalFileSystem> <begin Win32LocalFileSystem> package org.apache.hadoop.fs; /**************************************************************** Implement the FileSystem API for the checksumed local filesystem. * *****************************************************************/ public class Win32LocalFileSystem extends LocalFileSystem { public Win32LocalFileSystem() { this(new Win32RawLocalFileSystem()); } public Win32LocalFileSystem(FileSystem rawLocalFileSystem) { super(rawLocalFileSystem); rfs = rawLocalFileSystem; } } <end Win32LocalFileSystem> For a quickest compile, just manually compile and add these classes into the hadoop-core-1.0.4.jar (with this jar on the classpath) in the same order as above (backup your jar file first to be safe). Then edit the core-site.xml configuration file with the following entry: <property> <name>fs.file.impl</name> <value>org.apache.hadoop.fs.Win32LocalFileSystem</value> </property> All done - should work now.
          Hide
          Matt Giroux added a comment -

          Correction to my original post: org.apache.hadoop.fs.FileUtil calls the standard java.io.File setReadable, setWritable, setExecutable methods which always fail on Windows.

          Show
          Matt Giroux added a comment - Correction to my original post: org.apache.hadoop.fs.FileUtil calls the standard java.io.File setReadable, setWritable, setExecutable methods which always fail on Windows.
          Hide
          Todd Fast added a comment -

          Matt, the patch I created last year should fix the issue without needing to modify the Hadoop jars:

          https://github.com/congainc/patch-hadoop_7682-1.0.x-win

          Show
          Todd Fast added a comment - Matt, the patch I created last year should fix the issue without needing to modify the Hadoop jars: https://github.com/congainc/patch-hadoop_7682-1.0.x-win
          Hide
          Joshua Caplan added a comment -

          While we're creating a WinLocalFileSystem object, I found another helpful override:

          public boolean rename( Path src, Path dst ) throws IOException

          { if ( exists( dst ) ) delete( dst, true ); return super.rename( src, dst ); }

          This solves the problem with DistributedCache on Windows for local testing wherein a cache file expected at

          /tmp/hadoop-user/mapred/local/archive/random1_random2_random3/path/lastPath/file

          ends up at

          /tmp/hadoop-user/mapred/local/archive/random1_random2_random3/path/lastPath/lastPath-work--random4/file

          and is invisible to the tasks that need it. Can't even express how much pain that alleviates.

          Show
          Joshua Caplan added a comment - While we're creating a WinLocalFileSystem object, I found another helpful override: public boolean rename( Path src, Path dst ) throws IOException { if ( exists( dst ) ) delete( dst, true ); return super.rename( src, dst ); } This solves the problem with DistributedCache on Windows for local testing wherein a cache file expected at /tmp/hadoop-user/mapred/local/archive/random1_random2_random3/path/lastPath/file ends up at /tmp/hadoop-user/mapred/local/archive/random1_random2_random3/path/lastPath/lastPath-work--random4/file and is invisible to the tasks that need it. Can't even express how much pain that alleviates.
          Hide
          Magic Xie added a comment -

          Hi all, this bug seems fixed.
          Please see http://svn.apache.org/viewvc/hadoop/common/tags/release-2.1.0-beta-rc1/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java?revision=1065901&view=markup
          and HADOOP-7126.
          Fix file permission setting for RawLocalFileSystem on Windows. Contributed by Po Cheung.

          Show
          Magic Xie added a comment - Hi all, this bug seems fixed. Please see http://svn.apache.org/viewvc/hadoop/common/tags/release-2.1.0-beta-rc1/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java?revision=1065901&view=markup and HADOOP-7126 . Fix file permission setting for RawLocalFileSystem on Windows. Contributed by Po Cheung.

            People

            • Assignee:
              Unassigned
              Reporter:
              Magic Xie
            • Votes:
              27 Vote for this issue
              Watchers:
              34 Start watching this issue

              Dates

              • Created:
                Updated:

                Development